ROBERT A, SLOAN
The context: Why is this problem important to solve?
The objectives: What is the intended goal?
The key questions: What are the key questions that need to be answered?
The problem formulation: What are we trying to solve using data science?
The data set consists of 3 folders, i.e., 'test', 'train', and 'validation'. Each of these folders has four subfolders:
‘happy’: Images of people who have happy facial expressions.
‘sad’: Images of people with sad or upset facial expressions.
‘surprise’: Images of people who have shocked or surprised facial expressions.
‘neutral’: Images of people showing no prominent emotion in their facial expression at all.
PROBLEM STATEMENT:
CONTEXT: Computer Vision has use cases across almost all domains, from self-driving and evaluating the driver's experience in more automated and self-driving vehicles to the customer service industry across many types of jobs. The objective here is to build a range of different CNNs and test them to see what architecture works best for the problem of taking in an image and being able to recognize that A) there is a person in the image, but then also determine the emotional state of the person in the image.
Key Questions:
CNNs: Are they the right type of model for this task?
If CNNs are the right type of model, how are they constructed, and what kind of depth to the convolution layers is necessary and useful?
What is the best level of accuracy and overall performance that we can achieve before the model starts to become overfitted?
Transfer learning: How much can be gained by using pretrained models that are open-sourced? These models have been trained at no small expense on a large amount of labeled data that's time-intensive to collect and label. To reproduce them would be outside of what we are currently able to afford for the task, so can we gain from the hard work done by others, and if so, to what level?
We are trying to take in pixels from an image, value the pixel intensity, and then construct a neural network that can learn to see the world in terms of pixel values but be able to recognize the emergent features beyond that of just the pixel value. To see the human face and identify that one face has ears, and where there are ears, there is probably a face when looking at other images. Building on that to recognize the mouth and the facial features that change during different emotional states.
gpu_info = !nvidia-smi
gpu_info = '\n'.join(gpu_info)
if gpu_info.find('failed') >= 0:
print('Not connected to a GPU')
else:
print(gpu_info)
Thu Feb 22 18:20:41 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 53C P0 28W / 70W | 14077MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
!pip install tensorflow
!pip install keras-tuner
!pip install pandas
!pip install numpy
!pip install matplotlib
!pip install scikit-learn
!pip install imbalanced-learn
Requirement already satisfied: tensorflow in /usr/local/lib/python3.10/dist-packages (2.15.0)
Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.4.0)
Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.6.3)
Requirement already satisfied: flatbuffers>=23.5.26 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (23.5.26)
Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.5.4)
Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.2.0)
Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.9.0)
Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (16.0.6)
Requirement already satisfied: ml-dtypes~=0.2.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.2.0)
Requirement already satisfied: numpy<2.0.0,>=1.23.5 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.25.2)
Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from tensorflow) (23.2)
Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.20.3)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow) (67.7.2)
Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.16.0)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.4.0)
Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (4.9.0)
Requirement already satisfied: wrapt<1.15,>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.14.1)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.36.0)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.60.1)
Requirement already satisfied: tensorboard<2.16,>=2.15 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.15.2)
Requirement already satisfied: tensorflow-estimator<2.16,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.15.0)
Requirement already satisfied: keras<2.16,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.15.0)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow) (0.42.0)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (2.27.0)
Requirement already satisfied: google-auth-oauthlib<2,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (1.2.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (3.5.2)
Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (2.31.0)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (3.0.1)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (5.3.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (0.3.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (4.9)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow) (1.3.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (2024.2.2)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard<2.16,>=2.15->tensorflow) (2.1.5)
Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (0.5.1)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow) (3.2.2)
Collecting keras-tuner
Downloading keras_tuner-1.4.6-py3-none-any.whl (128 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 128.9/128.9 kB 4.3 MB/s eta 0:00:00
Requirement already satisfied: keras in /usr/local/lib/python3.10/dist-packages (from keras-tuner) (2.15.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from keras-tuner) (23.2)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from keras-tuner) (2.31.0)
Collecting kt-legacy (from keras-tuner)
Downloading kt_legacy-1.0.5-py3-none-any.whl (9.6 kB)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->keras-tuner) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->keras-tuner) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->keras-tuner) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->keras-tuner) (2024.2.2)
Installing collected packages: kt-legacy, keras-tuner
Successfully installed keras-tuner-1.4.6 kt-legacy-1.0.5
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (1.5.3)
Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2023.4)
Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.10/dist-packages (from pandas) (1.25.2)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas) (1.16.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (1.25.2)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (3.7.1)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.2.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (4.49.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.4.5)
Requirement already satisfied: numpy>=1.20 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.25.2)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (23.2)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (9.4.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (3.1.1)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (2.8.2)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (1.2.2)
Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.25.2)
Requirement already satisfied: scipy>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.11.4)
Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.3.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (3.3.0)
Requirement already satisfied: imbalanced-learn in /usr/local/lib/python3.10/dist-packages (0.10.1)
Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (1.25.2)
Requirement already satisfied: scipy>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (1.11.4)
Requirement already satisfied: scikit-learn>=1.0.2 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (1.2.2)
Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (1.3.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (3.3.0)
NOTE: Please use Google Colab from your browser for this notebook. Google.colab is NOT a library that can be downloaded locally on your device.
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
import tensorflow as tf
import zipfile
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import PIL
# Library for creating data paths
import os
from IPython.display import HTML
# Library for randomly selecting data points
import random
# Library for creating and showing plots
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
# Importing Deep Learning Libraries
from tensorflow.keras.utils import load_img
from tensorflow.keras.preprocessing.image import load_img, img_to_array
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.layers import Dense, Input, Dropout, GlobalAveragePooling2D, Flatten, Conv2D, BatchNormalization, Activation, MaxPooling2D, LeakyReLU
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.optimizers import Adam, SGD, RMSprop, Adamax, AdamW
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau
from tensorflow.keras.utils import to_categorical
# Importing all the required sub-modules from Keras
from keras.applications.vgg16 import VGG16
from tensorflow.keras.callbacks import LearningRateScheduler
from sklearn.utils import resample
from imblearn.over_sampling import SMOTE
path = '/content/drive/My Drive/Deep learning/CapstoneProject/Facial_emotion_images.zip'
with zipfile.ZipFile(path, 'r') as zip_ref:
zip_ref.extractall()
picture_size = 48
folder_path = "Facial_emotion_images/"
Note:
Let's look at our classes.
Write down your observation for each class. What do you think can be a unique feature of each emotion, that separates it from the remaining classes?
folder_path = 'Facial_emotion_images/'
num_happy = len(os.listdir(folder_path + "train/" + 'happy'))
# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 25px; padding: 10px; border-radius: 10px;'>" \
f"<h3 style='color: #333;'>Training Data Information:</h3>" \
f"<p style='color: blue;'>Number of training images for class <b>'happy'</b> is: <b>{num_happy}</b></p>" \
f"</div>"
# Display the styled output
display(HTML(output_html))
expression1 = 'happy'
plt.figure(figsize= (30,4))
for i in range(1, 21, 1):
plt.subplot(2, 10, i)
img = load_img(folder_path + "train/" + expression1 + "/" +
os.listdir(folder_path + "train/" + expression1)[i], target_size = (picture_size, picture_size))
# Seperate the two expressions with a horizontal line
if i == 8:
plt.axhline()
plt.imshow(img)
Number of training images for class 'happy' is: 3976
Observations and Insights:__
folder_path = 'Facial_emotion_images/'
num_surprise = len(os.listdir(folder_path + "train/" + 'surprise'))
num_happy= len(os.listdir(folder_path + "train/" + 'happy'))
num_neutral = len(os.listdir(folder_path + "train/" + 'neutral'))
num_sad = len(os.listdir(folder_path + "train/" + 'sad'))
# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; padding: 10px;text-align: center; font-size: 25px; border-radius: 10px;'>" \
f"<h3 style='color: #333;'>Training Data Information:</h3>" \
f"<p style='color: blue;'>Number of training images for class <b>'surprise'</b> is: <b>{num_surprise}</b></p>" \
f"</div>"
# Display the styled output
display(HTML(output_html))
expression = 'surprise'
plt.figure(figsize= (30
,4))
for i in range(1, 21, 1):
plt.subplot(2, 10, i)
img = load_img(folder_path + "train/" + expression + "/" +
os.listdir(folder_path + "train/" + expression)[i], target_size = (picture_size, picture_size))
plt.imshow(img)
plt.show()
Number of training images for class 'surprise' is: 3173
Observations and Insights:__
from IPython.display import HTML
import os
folder_path = 'Facial_emotion_images/'
num_sad = len(os.listdir(folder_path + "train/" + 'sad'))
# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 25px; padding: 10px; border-radius: 10px;'>" \
f"<h3 style='color: #333;'>Training Data Information:</h3>" \
f"<p style='color: blue;'>Number of training images for class <b>'sad'</b> is: <b>{num_sad}</b></p>" \
f"</div>"
display(HTML(output_html))
expression = 'sad'
plt.figure(figsize= (30,4))
for i in range(1, 21, 1):
plt.subplot(2, 10, i)
img = load_img(folder_path + "train/" + expression + "/" +
os.listdir(folder_path + "train/" + expression)[i], target_size = (picture_size, picture_size))
plt.imshow(img)
plt.show()
Number of training images for class 'sad' is: 3982
Observations and Insights:__
folder_path = 'Facial_emotion_images/'
num_neutral = len(os.listdir(folder_path + "train/" + 'neutral'))
# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 25px; padding: 10px; border-radius: 10px;'>" \
f"<h3 style='color: #333;'>Training Data Information:</h3>" \
f"<p style='color: blue;'>Number of training images for class <b>'neutral'</b> is: <b>{num_neutral}</b></p>" \
f"</div>"
# Display the styled output
display(HTML(output_html))
expression = 'neutral'
plt.figure(figsize= (30,4))
for i in range(1, 21, 1):
plt.subplot(2, 10, i)
img = load_img(folder_path + "train/" + expression + "/" +
os.listdir(folder_path + "train/" + expression)[i], target_size = (picture_size, picture_size))
plt.imshow(img)
plt.show()
Number of training images for class 'neutral' is: 3978
Observations and Insights:__
import os
from keras.preprocessing.image import load_img
import matplotlib.pyplot as plt
# Assuming 'base_dir' is correctly set to your dataset directory
base_dir = 'Facial_emotion_images/train/'
categories = ['happy', 'sad', 'neutral', 'surprise']
fig, ax = plt.subplots(nrows=len(categories), ncols=4, figsize=(30, 8)) # Adjusted figsize for better visibility
for i, category in enumerate(categories):
category_dir = os.path.join(base_dir, category)
sample_images = os.listdir(category_dir)[:4] # Get first 4 images of each category
for j, image in enumerate(sample_images):
img_path = os.path.join(category_dir, image)
img = load_img(img_path, target_size=(48, 48), color_mode='grayscale')
ax[i, j].imshow(img, cmap='gray')
ax[i, j].axis('off')
if j == 0: # Mark the category name more prominently
ax[i, j].set_ylabel(category.capitalize(), fontsize=14, labelpad=10, color='blue')
# Enhance layout and visibility
plt.tight_layout(pad=3.0)
plt.show()
folder_path = 'Facial_emotion_images/'
num_surprise = len(os.listdir(folder_path + "train/" + 'surprise'))
num_happy= len(os.listdir(folder_path + "train/" + 'happy'))
num_neutral = len(os.listdir(folder_path + "train/" + 'neutral'))
num_sad = len(os.listdir(folder_path + "train/" + 'sad'))
# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; padding: 10px;text-align: center; font-size: 25px; border-radius: 10px;'>" \
f"<h3 style='color: #333;'>Training Data Information:</h3>" \
f"<p style='color: blue;'>Number of training images for class <b>'surprise'</b> is: <b>{num_surprise}</b></p>" \
f"<p style='color: green;'>Number of training images for class <b>'neutral'</b> is: <b>{num_neutral}</b></p>" \
f"<p style='color: orange;'>Number of training images for class <b>'happy'</b> is: <b>{num_happy}</b></p>" \
f"<p style='color: red;'>Number of training images for class <b>'sad'</b> is: <b>{num_sad}</b></p>" \
f"</div>"
# Display the styled output
display(HTML(output_html))
# Code to plot histogram
plt.figure(figsize = (30, 7))
# Data
classes = ['surprise', 'happy', 'neutral', 'sad']
num_images = [3173, 3976, 3978, 3982]
%matplotlib inline
plt.subplot(2, 1, 1),
plt.bar(classes, num_images, color=['blue', 'orange', 'green', 'red'])
plt.xlabel('Class')
plt.ylabel('Number of Images')
plt.title('Number of Training Images per Class')
plt.show()
Number of training images for class 'surprise' is: 3173
Number of training images for class 'neutral' is: 3978
Number of training images for class 'happy' is: 3976
Number of training images for class 'sad' is: 3982
Think About It:
OBSERVATION: The categorys sad neutral and happy all have the same ammount of images however the suprise category has less.
Observations and Insights:__
The data set is skewed and needs to have some data augmentation methods applied to equalize the data set.
classes = ['surprise', 'happy', 'neutral', 'sad']
num_images = [3173, 3976, 3978, 3982]
plt.figure(figsize=(5, 5))
plt.pie(num_images, labels=classes, autopct='%1.1f%%', startangle=140, colors=['blue', 'orange', 'green', 'red'])
plt.title('Proportion of Each Class in the Dataset')
plt.show()
# Check the Data Set for imbalance
min_class_size = min(num_images)
max_class_size = max(num_images)
imbalance_ratio = max_class_size / min_class_size
import matplotlib.pyplot as plt
# Data
classes = ['surprise', 'happy', 'neutral', 'sad']
num_images = [3173, 3976, 3978, 3982]
%matplotlib inline
# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 25px; padding: 10px; border-radius: 10px;'>" \
f"<h3 style='color: red'>Imbalance Ratio: Categorical representation in the data set (Max Class Size / Min Class Size): 1.25 :</h3>" \
f"<p style='color: #666;'><b>''</b>Imbalance Ratio between classes is: <b>{imbalance_ratio:.2f}</b></p>" \
f"<p style='color: red;'><b>''</b> Sad image representation is: <b> 26.4% </b></p>" \
f"<p style='color: green;'><b>''</b>Neutral image representation is: <b> 26.3% </b></p>" \
f"<p style='color: orange;'><b>''</b> Happy image representation is: <b> 26.3% </b></p>" \
f"<p style='color: blue;'><b>''</b>Surprise image representation is: <b> 21.0% </b></p>" \
f"</div>"
# Display the styled output
display(HTML(output_html))
''Imbalance Ratio between classes is: 1.25
'' Sad image representation is: 26.4%
''Neutral image representation is: 26.3%
'' Happy image representation is: 26.3%
''Surprise image representation is: 21.0%
In the initial EDA of the dataset, the categories of emotions appeared to be balanced except for 'surprise'. Digging into the statistical significance of the disparity, I have concluded the following
An imbalance ratio of 1.25 (Max Class Size / Min Class Size) suggests a relatively mild imbalance among the classes in the dataset.
Impact of Mild Imbalance Model Performance: Model Performance Modern neural networks, particularly those used for image classification tasks (like CNNs), are quite robust to mild class imbalances. An imbalance ratio close to 1 indicates that the classes are fairly well represented.
Learning Dynamics: The slight imbalance might not significantly skew the learning process, meaning the model can still learn to generalize well across all classes without heavy reliance on data balancing techniques.
To make sure that the mild inbalance is not going to impact the training process the following steps have been employed in the following models traing. Monitor Class-specific Metrics: Even with a mild imbalance, monitoring metrics like precision, recall, and F1-score for each class is good practice. This ensures that all classes are correctly learned by the model, and no class is systematically favored or ignored.
Mild Data Augmentation: While aggressive oversampling or synthetic data generation might not be necessary, employing data augmentation techniques rotation, flipping, scaling on the training set, can further help handle the slight imbalance's impact.
In this section, we are creating data loaders that we will use as inputs to our Neural Network.
You have two options for the color_mode. You can set it to color_mode = 'rgb' or color_mode = 'grayscale'. You will need to try out both and see for yourself which one gives better performance.
from keras.preprocessing.image import ImageDataGenerator
# Define image size and paths
picture_size = 48
base_path = "Facial_emotion_images/"
train_path = base_path + "train/"
validation_path = base_path + "validation/"
test_path = base_path + "test/"
# Initialize the data generator for training data with data augmentation
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=False,
fill_mode='nearest'
)
# For validation and test sets, usually only rescaling is applied
test_val_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(
train_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical'
)
validation_generator = test_val_datagen.flow_from_directory(
validation_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical'
)
test_generator = test_val_datagen.flow_from_directory(
test_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical',
shuffle=False # Keep data in order for evaluation
)
# Getting the total Training images, Testing images and Validation images.
traingen = train_generator.n
testgen = test_generator.n
validationgen = validation_generator.n
# Using inline HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 25px; padding: 10px; border-radius: 10px;'>" \
f"<p style='color: red;'><b>''</b>Total Training Images <b> {traingen}</b></p>" \
f"<p style='color: black;'><b>''</b>Total Testing Images <b> {validationgen} </b></p>" \
f"<p style='color: blue;'><b>'Data Set'</b>Total Validation Images <b>{testgen} </b></p>" \
f"</div>"
# Display the styled output
display(HTML(output_html))
#Defining the function for creating the custom classification report
def metrics_score(actual, predicted):
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
print(classification_report(actual, predicted))
cm = confusion_matrix(actual, predicted)
plt.figure(figsize = (8, 5))
sns.heatmap(cm, annot = True, fmt = '.0f', xticklabels = class_names_list, yticklabels = class_names_list)
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import confusion_matrix, classification_report
def plot_model_performance_and_confusion_matrix(model, history, test_generator, metrics_function=None):
# Plot training and validation accuracy
plt.figure(figsize=(25, 5))
plt.subplot(1, 3, 1)
plt.plot(history.history['accuracy'], label='Training Accuracy')
plt.plot(history.history['val_accuracy'], label='Validation Accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.title('Model Accuracy')
plt.legend()
# Plot training and validation loss
plt.subplot(1, 3, 2)
plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Model Loss')
plt.legend()
# Predict labels for the test set
predicted_labels = np.argmax(model.predict(test_generator), axis=1)
# Get the true labels for the test set
true_labels = test_generator.classes
# Optionally print class names
class_names_list = list(test_generator.class_indices.keys())
print("Class names in the dataset:", class_names_list)
# Optionally calculate and display metrics
if metrics_function is not None:
metrics_function(true_labels, predicted_labels)
# Plot the confusion matrix
cm = confusion_matrix(true_labels, predicted_labels)
plt.subplot(1, 3, 3)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", xticklabels=class_names_list, yticklabels=class_names_list)
plt.xlabel("Predicted Label")
plt.ylabel("True Label")
plt.title("Confusion Matrix")
plt.show()
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import confusion_matrix, classification_report
def plot_model_stats(model, history, test_generator, model_name):
# Compute confusion matrix
cm = confusion_matrix(test_generator.classes, np.argmax(model.predict(test_generator), axis=1))
# Plot the training and validation accuracy values
plt.figure(figsize=(15, 5))
plt.subplot(1, 3, 1)
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title(f'{model_name} Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Plot the training and validation loss values
plt.subplot(1, 3, 2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title(f'{model_name} Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Plot confusion matrix
plt.subplot(1, 3, 3)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", cbar=False)
plt.xlabel("Predicted Label")
plt.ylabel("True Label")
plt.title(f"Confusion Matrix for {model_name}")
plt.tight_layout() # Adjust layout to not overlap figures
plt.show()
def metrics_score(actual, predicted):
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
print(classification_report(actual, predicted))
cm = confusion_matrix(actual, predicted)
plt.figure(figsize = (8, 5))
sns.heatmap(cm, annot = True, fmt = '.0f', xticklabels = class_names_list, yticklabels = class_names_list)
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
import os
from IPython.display import HTML, display
base_path = "Facial_emotion_images/"
subsets = ['train', 'test', 'validation']
classes = ['happy', 'sad', 'neutral', 'surprise']
subset_counts = {subset: 0 for subset in subsets}
for subset in subsets:
for emotion in classes:
# Dir construction
dir_path = os.path.join(base_path, subset, emotion)
# Count files in directories and add to subset count
if os.path.exists(dir_path):
files_count = len([name for name in os.listdir(dir_path) if os.path.isfile(os.path.join(dir_path, name))])
subset_counts[subset] += files_count
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 10px; border-radius: 10px;'>" \
f"<p style='color: red;'><b>Total Training Images:</b> {subset_counts['train']}</p>" \
f"<p style='color: black;'><b>Total Testing Images:</b> {subset_counts['test']} </p>" \
f"<p style='color: blue;'><b>Total Validation Images:</b> {subset_counts['validation']} </p>" \
f"</div>"
# Display the styled output
display(HTML(output_html))
Total Training Images: 15109
Total Testing Images: 128
Total Validation Images: 4977
import os
from IPython.display import HTML, display
base_path = "Facial_emotion_images/"
subsets = ['train', 'test', 'validation']
classes = ['happy', 'sad', 'neutral', 'surprise']
class_counts = {emotion: 0 for emotion in classes}
for subset in subsets:
for emotion in classes:
dir_path = os.path.join(base_path, subset, emotion)
if os.path.exists(dir_path):
class_counts[emotion] += len([name for name in os.listdir(dir_path) if os.path.isfile(os.path.join(dir_path, name))])
# Generate HTML content
output_html = "<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>"
output_html += "<h3 style='color: #333;'>Class Distribution</h3>"
for emotion, count in class_counts.items():
output_html += f"<p style='color: #333;'><b>{emotion.capitalize()}:</b> {count}</p>"
output_html += "</div>"
# Display the styled output
display(HTML(output_html))
Happy: 5833
Sad: 5153
Neutral: 5226
Surprise: 4002
from sklearn.utils.class_weight import compute_class_weight
import numpy as np
from IPython.display import HTML, display
total_samples = sum(class_counts.values())
n_classes = len(classes)
# Compute class weights using sklearn's compute_class_weight function for better accuracy
y_integers = np.array([classes.index(emotion) for emotion in class_counts.keys() for _ in range(class_counts[emotion])])
class_weights = compute_class_weight(class_weight='balanced', classes=np.unique(y_integers), y=y_integers)
class_weights_dict = {classes[i]: weight for i, weight in enumerate(class_weights)}
# List the class weights in styled HTML
output_html = "<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>"
output_html += "<h3 style='color: #333;'>Computed Class Weights</h3>"
for emotion, weight in class_weights_dict.items():
output_html += f"<p style='color: #333;'><b>{emotion.capitalize()}:</b> {weight:.2f}</p>"
output_html += "</div>"
# Display the styled output
display(HTML(output_html))
Happy: 0.87
Sad: 0.98
Neutral: 0.97
Surprise: 1.26
from sklearn.utils.class_weight import compute_class_weight
import numpy as np
class_labels = np.array([classes.index(cls) for cls in classes for _ in range(class_counts[cls])])
class_weights = compute_class_weight(class_weight='balanced', classes=np.unique(class_labels), y=class_labels)
class_weights_dict = {i: weight for i, weight in enumerate(class_weights)}
class_labels = np.array([classes.index(cls) for cls in classes for _ in range(class_counts[cls])])
class_weights = compute_class_weight(class_weight='balanced', classes=np.unique(class_labels), y=class_labels)
class_weights_dict = {i: weight for i, weight in enumerate(class_weights)}
class_weights_dict = {classes[i]: weight for i, weight in enumerate(class_weights)}
print("Class labels:", class_labels)
print("Class weights dictionary:", class_weights_dict)
print("Class weights:", class_weights)
# Adjust class weights to sum to 1
total_weight = sum(class_weights)
class_weights_adjusted = {cls: weight / total_weight for cls, weight in class_weights_dict.items()}
print("Class weights adjusted:", class_weights_adjusted)
Class labels: [0 0 0 ... 3 3 3]
Class weights dictionary: {'surprise': 1.262743628185907, 'happy': 0.8663637922166981, 'neutral': 0.96699196326062, 'sad': 0.9806908596933825}
Class weights: [1.26274363 0.86636379 0.96699196 0.98069086]
Class weights adjusted: {'surprise': 0.30973966105899836, 'happy': 0.21251125039569885, 'neutral': 0.23719443619558198, 'sad': 0.24055465234972084}
classes_sorted = sorted(classes)
class_weights_dict = {cls: weight for cls, weight in zip(classes_sorted, class_weights)}
# Adjusting ClassWeights to ensure proper indicies assignment
class_weights_adjusted = {train_generator.class_indices[cls]: weight for cls, weight in class_weights_dict.items()}
Think About It:
import tensorflow as tf
# Fixing the random seed generator to ensure random initilization numbers
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, GlobalAveragePooling2D
from tensorflow.keras.layers import Flatten, Dense, Dropout, BatchNormalization
from tensorflow.keras.optimizers import Adam
cnn_model_1 = Sequential([
# First Convolutional Block
Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(48, 48, 3)),
BatchNormalization(),
Conv2D(64, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.25),
# Second Convolutional Block
Conv2D(128, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(128, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.5),
# Third Convolutional Block
Conv2D(256, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(256, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(256, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.5),
# Fourth Convolutional Block
Conv2D(1024, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(1024, (3, 3), activation='relu', padding='same'),
GlobalAveragePooling2D(),
(Flatten()),
# Dense Layers
Dense(1024, activation='relu'),
Dense(256, activation='relu'),
Dropout(0.3),
# Output
Dense(128, activation='relu'),
Dropout(0.1),
Dense(64, activation='relu'),
Dropout(0.25),
Dense(4, activation='softmax')
])
cnn_model_summary = cnn_model_1.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 48, 48, 64) 1792
batch_normalization (Batch (None, 48, 48, 64) 256
Normalization)
conv2d_1 (Conv2D) (None, 48, 48, 64) 36928
max_pooling2d (MaxPooling2 (None, 24, 24, 64) 0
D)
dropout (Dropout) (None, 24, 24, 64) 0
conv2d_2 (Conv2D) (None, 24, 24, 128) 73856
batch_normalization_1 (Bat (None, 24, 24, 128) 512
chNormalization)
conv2d_3 (Conv2D) (None, 24, 24, 128) 147584
max_pooling2d_1 (MaxPoolin (None, 12, 12, 128) 0
g2D)
dropout_1 (Dropout) (None, 12, 12, 128) 0
conv2d_4 (Conv2D) (None, 12, 12, 256) 295168
batch_normalization_2 (Bat (None, 12, 12, 256) 1024
chNormalization)
conv2d_5 (Conv2D) (None, 12, 12, 256) 590080
batch_normalization_3 (Bat (None, 12, 12, 256) 1024
chNormalization)
conv2d_6 (Conv2D) (None, 12, 12, 256) 590080
max_pooling2d_2 (MaxPoolin (None, 6, 6, 256) 0
g2D)
dropout_2 (Dropout) (None, 6, 6, 256) 0
conv2d_7 (Conv2D) (None, 6, 6, 1024) 2360320
batch_normalization_4 (Bat (None, 6, 6, 1024) 4096
chNormalization)
conv2d_8 (Conv2D) (None, 6, 6, 1024) 9438208
global_average_pooling2d ( (None, 1024) 0
GlobalAveragePooling2D)
flatten (Flatten) (None, 1024) 0
dense (Dense) (None, 1024) 1049600
dense_1 (Dense) (None, 256) 262400
dropout_3 (Dropout) (None, 256) 0
dense_2 (Dense) (None, 128) 32896
dropout_4 (Dropout) (None, 128) 0
dense_3 (Dense) (None, 64) 8256
dropout_5 (Dropout) (None, 64) 0
dense_4 (Dense) (None, 4) 260
=================================================================
Total params: 14894340 (56.82 MB)
Trainable params: 14890884 (56.80 MB)
Non-trainable params: 3456 (13.50 KB)
_________________________________________________________________
cnn_model_1.compile(optimizer=AdamW(learning_rate=0.0005), # Adjust learning rate as needed
loss='categorical_crossentropy',
metrics=['accuracy'])
from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau
early_stopping = EarlyStopping(monitor='accuracy', patience=5, verbose=1, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=3, verbose=1, min_lr=0.000002)
history = cnn_model_1.fit(
train_generator,
epochs=25, # Adjust based on early stopping
validation_data=validation_generator,
callbacks=[early_stopping, reduce_lr],
class_weight=class_weights_adjusted
)
Epoch 1/25 473/473 [==============================] - 33s 54ms/step - loss: 1.2893 - accuracy: 0.3542 - val_loss: 1.1666 - val_accuracy: 0.4744 - lr: 5.0000e-04 Epoch 2/25 473/473 [==============================] - 25s 52ms/step - loss: 1.0293 - accuracy: 0.5333 - val_loss: 0.9968 - val_accuracy: 0.5445 - lr: 5.0000e-04 Epoch 3/25 473/473 [==============================] - 25s 53ms/step - loss: 0.8731 - accuracy: 0.6343 - val_loss: 0.8473 - val_accuracy: 0.6383 - lr: 5.0000e-04 Epoch 4/25 473/473 [==============================] - 25s 53ms/step - loss: 0.7914 - accuracy: 0.6766 - val_loss: 0.9087 - val_accuracy: 0.6365 - lr: 5.0000e-04 Epoch 5/25 473/473 [==============================] - 25s 54ms/step - loss: 0.7443 - accuracy: 0.6993 - val_loss: 0.7194 - val_accuracy: 0.7079 - lr: 5.0000e-04 Epoch 6/25 473/473 [==============================] - 25s 54ms/step - loss: 0.7043 - accuracy: 0.7152 - val_loss: 0.7093 - val_accuracy: 0.7229 - lr: 5.0000e-04 Epoch 7/25 473/473 [==============================] - 25s 53ms/step - loss: 0.6734 - accuracy: 0.7295 - val_loss: 0.7230 - val_accuracy: 0.7163 - lr: 5.0000e-04 Epoch 8/25 473/473 [==============================] - 25s 53ms/step - loss: 0.6391 - accuracy: 0.7433 - val_loss: 0.7180 - val_accuracy: 0.7203 - lr: 5.0000e-04 Epoch 9/25 473/473 [==============================] - 25s 53ms/step - loss: 0.6195 - accuracy: 0.7541 - val_loss: 0.6271 - val_accuracy: 0.7511 - lr: 5.0000e-04 Epoch 10/25 473/473 [==============================] - 25s 53ms/step - loss: 0.5876 - accuracy: 0.7668 - val_loss: 0.6473 - val_accuracy: 0.7396 - lr: 5.0000e-04 Epoch 11/25 473/473 [==============================] - 25s 54ms/step - loss: 0.5608 - accuracy: 0.7767 - val_loss: 0.6750 - val_accuracy: 0.7326 - lr: 5.0000e-04 Epoch 12/25 472/473 [============================>.] - ETA: 0s - loss: 0.5481 - accuracy: 0.7880 Epoch 12: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513. 473/473 [==============================] - 25s 54ms/step - loss: 0.5475 - accuracy: 0.7883 - val_loss: 0.6758 - val_accuracy: 0.7324 - lr: 5.0000e-04 Epoch 13/25 473/473 [==============================] - 25s 53ms/step - loss: 0.4534 - accuracy: 0.8245 - val_loss: 0.6064 - val_accuracy: 0.7816 - lr: 1.0000e-04 Epoch 14/25 473/473 [==============================] - 25s 53ms/step - loss: 0.4201 - accuracy: 0.8327 - val_loss: 0.6151 - val_accuracy: 0.7824 - lr: 1.0000e-04 Epoch 15/25 473/473 [==============================] - 25s 54ms/step - loss: 0.3974 - accuracy: 0.8439 - val_loss: 0.6472 - val_accuracy: 0.7746 - lr: 1.0000e-04 Epoch 16/25 472/473 [============================>.] - ETA: 0s - loss: 0.3793 - accuracy: 0.8480 Epoch 16: ReduceLROnPlateau reducing learning rate to 2.0000000949949027e-05. 473/473 [==============================] - 25s 54ms/step - loss: 0.3794 - accuracy: 0.8479 - val_loss: 0.6692 - val_accuracy: 0.7752 - lr: 1.0000e-04 Epoch 17/25 473/473 [==============================] - 25s 53ms/step - loss: 0.3476 - accuracy: 0.8654 - val_loss: 0.6500 - val_accuracy: 0.7808 - lr: 2.0000e-05 Epoch 18/25 473/473 [==============================] - 25s 53ms/step - loss: 0.3418 - accuracy: 0.8661 - val_loss: 0.6576 - val_accuracy: 0.7830 - lr: 2.0000e-05 Epoch 19/25 473/473 [==============================] - ETA: 0s - loss: 0.3340 - accuracy: 0.8690 Epoch 19: ReduceLROnPlateau reducing learning rate to 4.000000262749381e-06. 473/473 [==============================] - 25s 54ms/step - loss: 0.3340 - accuracy: 0.8690 - val_loss: 0.6585 - val_accuracy: 0.7840 - lr: 2.0000e-05 Epoch 20/25 473/473 [==============================] - 25s 54ms/step - loss: 0.3277 - accuracy: 0.8707 - val_loss: 0.6671 - val_accuracy: 0.7822 - lr: 4.0000e-06 Epoch 21/25 473/473 [==============================] - 25s 54ms/step - loss: 0.3277 - accuracy: 0.8656 - val_loss: 0.6694 - val_accuracy: 0.7824 - lr: 4.0000e-06 Epoch 22/25 472/473 [============================>.] - ETA: 0s - loss: 0.3257 - accuracy: 0.8731 Epoch 22: ReduceLROnPlateau reducing learning rate to 2e-06. 473/473 [==============================] - 25s 54ms/step - loss: 0.3255 - accuracy: 0.8732 - val_loss: 0.6722 - val_accuracy: 0.7820 - lr: 4.0000e-06 Epoch 23/25 473/473 [==============================] - 25s 53ms/step - loss: 0.3227 - accuracy: 0.8726 - val_loss: 0.6737 - val_accuracy: 0.7816 - lr: 2.0000e-06 Epoch 24/25 473/473 [==============================] - 25s 53ms/step - loss: 0.3273 - accuracy: 0.8723 - val_loss: 0.6743 - val_accuracy: 0.7820 - lr: 2.0000e-06 Epoch 25/25 473/473 [==============================] - 25s 53ms/step - loss: 0.3227 - accuracy: 0.8707 - val_loss: 0.6747 - val_accuracy: 0.7820 - lr: 2.0000e-06
history = cnn_model_1.fit(
train_generator,
epochs=25, # Adjust based on early stopping
validation_data=validation_generator,
callbacks=[early_stopping, reduce_lr],
class_weight=class_weights_adjusted
)
Epoch 1/25 473/473 [==============================] - 27s 46ms/step - loss: 1.3672 - accuracy: 0.3026 - val_loss: 1.3214 - val_accuracy: 0.2648 - lr: 5.0000e-04 Epoch 2/25 473/473 [==============================] - 22s 46ms/step - loss: 1.2208 - accuracy: 0.3796 - val_loss: 1.2417 - val_accuracy: 0.3122 - lr: 5.0000e-04 Epoch 3/25 473/473 [==============================] - 22s 45ms/step - loss: 1.1498 - accuracy: 0.4201 - val_loss: 1.2578 - val_accuracy: 0.3255 - lr: 5.0000e-04 Epoch 4/25 473/473 [==============================] - 21s 45ms/step - loss: 1.1108 - accuracy: 0.4621 - val_loss: 1.1026 - val_accuracy: 0.5041 - lr: 5.0000e-04 Epoch 5/25 473/473 [==============================] - 21s 45ms/step - loss: 1.0652 - accuracy: 0.4964 - val_loss: 1.0543 - val_accuracy: 0.5188 - lr: 5.0000e-04 Epoch 6/25 473/473 [==============================] - 21s 45ms/step - loss: 0.9929 - accuracy: 0.5599 - val_loss: 0.9972 - val_accuracy: 0.5821 - lr: 5.0000e-04 Epoch 7/25 473/473 [==============================] - 21s 44ms/step - loss: 0.9165 - accuracy: 0.6112 - val_loss: 1.0254 - val_accuracy: 0.5600 - lr: 5.0000e-04 Epoch 8/25 473/473 [==============================] - 21s 44ms/step - loss: 0.8532 - accuracy: 0.6508 - val_loss: 0.7439 - val_accuracy: 0.6994 - lr: 5.0000e-04 Epoch 9/25 473/473 [==============================] - 21s 44ms/step - loss: 0.8271 - accuracy: 0.6621 - val_loss: 0.7456 - val_accuracy: 0.7036 - lr: 5.0000e-04 Epoch 10/25 473/473 [==============================] - 21s 44ms/step - loss: 0.8005 - accuracy: 0.6705 - val_loss: 0.6997 - val_accuracy: 0.7117 - lr: 5.0000e-04 Epoch 11/25 473/473 [==============================] - 21s 44ms/step - loss: 0.7749 - accuracy: 0.6885 - val_loss: 0.6853 - val_accuracy: 0.7316 - lr: 5.0000e-04 Epoch 12/25 473/473 [==============================] - 21s 44ms/step - loss: 0.7588 - accuracy: 0.6870 - val_loss: 0.6973 - val_accuracy: 0.7151 - lr: 5.0000e-04 Epoch 13/25 473/473 [==============================] - 21s 44ms/step - loss: 0.7347 - accuracy: 0.7069 - val_loss: 0.6655 - val_accuracy: 0.7418 - lr: 5.0000e-04 Epoch 14/25 473/473 [==============================] - 21s 44ms/step - loss: 0.7219 - accuracy: 0.7054 - val_loss: 0.6480 - val_accuracy: 0.7448 - lr: 5.0000e-04 Epoch 15/25 473/473 [==============================] - 21s 44ms/step - loss: 0.7119 - accuracy: 0.7107 - val_loss: 0.6969 - val_accuracy: 0.7267 - lr: 5.0000e-04 Epoch 16/25 473/473 [==============================] - 21s 44ms/step - loss: 0.7008 - accuracy: 0.7190 - val_loss: 0.6514 - val_accuracy: 0.7338 - lr: 5.0000e-04 Epoch 17/25 472/473 [============================>.] - ETA: 0s - loss: 0.6900 - accuracy: 0.7220 Epoch 17: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513. 473/473 [==============================] - 21s 44ms/step - loss: 0.6894 - accuracy: 0.7222 - val_loss: 0.6523 - val_accuracy: 0.7509 - lr: 5.0000e-04 Epoch 18/25 473/473 [==============================] - 21s 44ms/step - loss: 0.6402 - accuracy: 0.7397 - val_loss: 0.5846 - val_accuracy: 0.7709 - lr: 1.0000e-04 Epoch 19/25 473/473 [==============================] - 21s 44ms/step - loss: 0.6118 - accuracy: 0.7498 - val_loss: 0.5854 - val_accuracy: 0.7687 - lr: 1.0000e-04 Epoch 20/25 473/473 [==============================] - 21s 44ms/step - loss: 0.6006 - accuracy: 0.7551 - val_loss: 0.5718 - val_accuracy: 0.7780 - lr: 1.0000e-04 Epoch 21/25 473/473 [==============================] - 21s 44ms/step - loss: 0.6130 - accuracy: 0.7484 - val_loss: 0.5883 - val_accuracy: 0.7675 - lr: 1.0000e-04 Epoch 22/25 473/473 [==============================] - 21s 44ms/step - loss: 0.5937 - accuracy: 0.7576 - val_loss: 0.5962 - val_accuracy: 0.7669 - lr: 1.0000e-04 Epoch 23/25 472/473 [============================>.] - ETA: 0s - loss: 0.5907 - accuracy: 0.7559 Epoch 23: ReduceLROnPlateau reducing learning rate to 2.0000000949949027e-05. 473/473 [==============================] - 21s 44ms/step - loss: 0.5912 - accuracy: 0.7557 - val_loss: 0.5798 - val_accuracy: 0.7673 - lr: 1.0000e-04 Epoch 24/25 473/473 [==============================] - 21s 44ms/step - loss: 0.5820 - accuracy: 0.7642 - val_loss: 0.5684 - val_accuracy: 0.7812 - lr: 2.0000e-05 Epoch 25/25 473/473 [==============================] - 21s 44ms/step - loss: 0.5749 - accuracy: 0.7667 - val_loss: 0.5712 - val_accuracy: 0.7788 - lr: 2.0000e-05
cnn_model_1.save('cnn_model_1bestscores.h5')
/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py:3103: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
saving_api.save_model(
Note: .h5 format is usable with python based backend such as flask however since the web is becoming dominated by javascript frame works such as react for front end client side user interfaces, and node.js backend and serverside functions converting the saved model to json format simplifys the deployment phase.
!tensorflowjs_converter --input_format=keras /content/cnn_model_1bestscores.h5 /content/recallprecisionf1js
2024-02-20 00:53:00.430837: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-02-20 00:53:00.430894: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-02-20 00:53:00.432410: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-02-20 00:53:01.465249: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
# Evaluate the model on the test set
val_loss, val_accuracy = cnn_model_1.evaluate(
test_generator,
steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
4/4 [==============================] - 0s 19ms/step - loss: 0.6674 - accuracy: 0.8047 Test Loss: 0.6673559546470642 Test Accuracy: 0.8046875
plot_model_performance_and_confusion_matrix(cnn_model_1, history, test_generator,
metrics_function=lambda true, pred: print(classification_report(true, pred)))
4/4 [==============================] - 0s 15ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
precision recall f1-score support
0 0.87 0.81 0.84 32
1 0.65 0.88 0.75 32
2 0.81 0.69 0.75 32
3 0.96 0.84 0.90 32
accuracy 0.80 128
macro avg 0.82 0.80 0.81 128
weighted avg 0.82 0.80 0.81 128
from IPython.display import HTML, display
# Assuming val_loss and val_accuracy are defined from your model's evaluation
# Evaluate the model on the test set
val_loss, val_accuracy = cnn_model_1.evaluate(
test_generator,
steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
# Format the evaluation results as HTML
output_html = f"""
<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>
<p style='color: red;'><b>Test Loss:</b> CNN_MODEL_1 </p>
<p style='color: red;'><b>Test Loss:</b> {val_loss:.4f}</p>
<p style='color: blue;'><b>Test Accuracy:</b> {val_accuracy:.4%} </p>
</div>
"""
# Display the styled output
display(HTML(output_html))
4/4 [==============================] - 0s 19ms/step - loss: 0.6674 - accuracy: 0.8047 Test Loss: 0.6673559546470642 Test Accuracy: 0.8046875
Test Loss: CNN_MODEL_1
Test Loss: 0.6674
Test Accuracy: 80.4688%
# Fixing the random seed generator to ensure random initilization numbers
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, GlobalAveragePooling2D
from tensorflow.keras.layers import Flatten, Dense, Dropout, BatchNormalization
# Define the CNN model
cnn_model_updated = Sequential([
# First Convolutional Block
Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(48, 48, 3)),
BatchNormalization(),
Conv2D(64, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.2), # Slightly reduced dropout
# Second Convolutional Block
Conv2D(128, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(128, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.3), # Adjusted dropout
# Third Convolutional Block
Conv2D(256, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(256, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.4), # Adjusted dropout
# Fourth Convolutional Block
Conv2D(512, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.4), # Keeping consistent dropout
# Global Average Pooling replaces Flatten to reduce parameters and potential overfitting
GlobalAveragePooling2D(),
Dropout(0.4), # Adjusted dropout
# Simplified Dense Layers
Dense(512, activation='relu'),
Dropout(0.4),
Dense(4, activation='softmax') # Output layer for 4 classes
])
# Model Summary
cnn_model_updated.summary()
cnn_model_updated.compile(optimizer=Adamax(learning_rate=0.001), # Adjusted learning rate
loss='categorical_crossentropy',
metrics=['accuracy'])
# Callbacks
from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau
early_stopping = EarlyStopping(monitor='val_accuracy', patience=5, verbose=1, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=3, verbose=1, min_lr=0.0001)
Model: "sequential_13"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_87 (Conv2D) (None, 48, 48, 64) 1792
batch_normalization_61 (Ba (None, 48, 48, 64) 256
tchNormalization)
conv2d_88 (Conv2D) (None, 48, 48, 64) 36928
max_pooling2d_48 (MaxPooli (None, 24, 24, 64) 0
ng2D)
dropout_71 (Dropout) (None, 24, 24, 64) 0
conv2d_89 (Conv2D) (None, 24, 24, 128) 73856
batch_normalization_62 (Ba (None, 24, 24, 128) 512
tchNormalization)
conv2d_90 (Conv2D) (None, 24, 24, 128) 147584
max_pooling2d_49 (MaxPooli (None, 12, 12, 128) 0
ng2D)
dropout_72 (Dropout) (None, 12, 12, 128) 0
conv2d_91 (Conv2D) (None, 12, 12, 256) 295168
batch_normalization_63 (Ba (None, 12, 12, 256) 1024
tchNormalization)
conv2d_92 (Conv2D) (None, 12, 12, 256) 590080
batch_normalization_64 (Ba (None, 12, 12, 256) 1024
tchNormalization)
max_pooling2d_50 (MaxPooli (None, 6, 6, 256) 0
ng2D)
dropout_73 (Dropout) (None, 6, 6, 256) 0
conv2d_93 (Conv2D) (None, 6, 6, 512) 1180160
batch_normalization_65 (Ba (None, 6, 6, 512) 2048
tchNormalization)
max_pooling2d_51 (MaxPooli (None, 3, 3, 512) 0
ng2D)
dropout_74 (Dropout) (None, 3, 3, 512) 0
global_average_pooling2d_1 (None, 512) 0
2 (GlobalAveragePooling2D)
dropout_75 (Dropout) (None, 512) 0
dense_37 (Dense) (None, 512) 262656
dropout_76 (Dropout) (None, 512) 0
dense_38 (Dense) (None, 4) 2052
=================================================================
Total params: 2595140 (9.90 MB)
Trainable params: 2592708 (9.89 MB)
Non-trainable params: 2432 (9.50 KB)
_________________________________________________________________
history = cnn_model_updated.fit(
train_generator,
epochs=50, # Adjust based on early stopping
validation_data=validation_generator,
callbacks=[early_stopping, reduce_lr],
class_weight=class_weights_adjusted
)
Epoch 1/50 473/473 [==============================] - 21s 39ms/step - loss: 1.5330 - accuracy: 0.3007 - val_loss: 1.5124 - val_accuracy: 0.2614 - lr: 0.0010 Epoch 2/50 473/473 [==============================] - 18s 38ms/step - loss: 1.3547 - accuracy: 0.3367 - val_loss: 1.3045 - val_accuracy: 0.4069 - lr: 0.0010 Epoch 3/50 473/473 [==============================] - 18s 37ms/step - loss: 1.2599 - accuracy: 0.3876 - val_loss: 1.2079 - val_accuracy: 0.4392 - lr: 0.0010 Epoch 4/50 473/473 [==============================] - 18s 39ms/step - loss: 1.1803 - accuracy: 0.4438 - val_loss: 1.0475 - val_accuracy: 0.5152 - lr: 0.0010 Epoch 5/50 473/473 [==============================] - 18s 38ms/step - loss: 1.1001 - accuracy: 0.5031 - val_loss: 1.1717 - val_accuracy: 0.5286 - lr: 0.0010 Epoch 6/50 473/473 [==============================] - 18s 38ms/step - loss: 0.9893 - accuracy: 0.5705 - val_loss: 0.8159 - val_accuracy: 0.6614 - lr: 0.0010 Epoch 7/50 473/473 [==============================] - 19s 41ms/step - loss: 0.9038 - accuracy: 0.6192 - val_loss: 0.8654 - val_accuracy: 0.6237 - lr: 0.0010 Epoch 8/50 473/473 [==============================] - 18s 38ms/step - loss: 0.8652 - accuracy: 0.6345 - val_loss: 0.8329 - val_accuracy: 0.6440 - lr: 0.0010 Epoch 9/50 473/473 [==============================] - 18s 38ms/step - loss: 0.8341 - accuracy: 0.6546 - val_loss: 0.6905 - val_accuracy: 0.7279 - lr: 0.0010 Epoch 10/50 473/473 [==============================] - 19s 40ms/step - loss: 0.8032 - accuracy: 0.6685 - val_loss: 0.7138 - val_accuracy: 0.7151 - lr: 0.0010 Epoch 11/50 473/473 [==============================] - 18s 39ms/step - loss: 0.7811 - accuracy: 0.6790 - val_loss: 0.7618 - val_accuracy: 0.6908 - lr: 0.0010 Epoch 12/50 473/473 [==============================] - ETA: 0s - loss: 0.7622 - accuracy: 0.6912 Epoch 12: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026. 473/473 [==============================] - 18s 39ms/step - loss: 0.7622 - accuracy: 0.6912 - val_loss: 0.8096 - val_accuracy: 0.6610 - lr: 0.0010 Epoch 13/50 473/473 [==============================] - 18s 38ms/step - loss: 0.7197 - accuracy: 0.7091 - val_loss: 0.6167 - val_accuracy: 0.7490 - lr: 2.0000e-04 Epoch 14/50 473/473 [==============================] - 18s 39ms/step - loss: 0.7033 - accuracy: 0.7167 - val_loss: 0.6098 - val_accuracy: 0.7553 - lr: 2.0000e-04 Epoch 15/50 473/473 [==============================] - 18s 39ms/step - loss: 0.6967 - accuracy: 0.7189 - val_loss: 0.6111 - val_accuracy: 0.7494 - lr: 2.0000e-04 Epoch 16/50 473/473 [==============================] - 19s 39ms/step - loss: 0.6935 - accuracy: 0.7200 - val_loss: 0.6152 - val_accuracy: 0.7468 - lr: 2.0000e-04 Epoch 17/50 473/473 [==============================] - 19s 41ms/step - loss: 0.6796 - accuracy: 0.7239 - val_loss: 0.5981 - val_accuracy: 0.7599 - lr: 2.0000e-04 Epoch 18/50 473/473 [==============================] - 19s 41ms/step - loss: 0.6750 - accuracy: 0.7243 - val_loss: 0.5996 - val_accuracy: 0.7577 - lr: 2.0000e-04 Epoch 19/50 473/473 [==============================] - 19s 40ms/step - loss: 0.6636 - accuracy: 0.7290 - val_loss: 0.6017 - val_accuracy: 0.7615 - lr: 2.0000e-04 Epoch 20/50 473/473 [==============================] - ETA: 0s - loss: 0.6692 - accuracy: 0.7312 Epoch 20: ReduceLROnPlateau reducing learning rate to 0.0001. 473/473 [==============================] - 20s 41ms/step - loss: 0.6692 - accuracy: 0.7312 - val_loss: 0.6029 - val_accuracy: 0.7631 - lr: 2.0000e-04 Epoch 21/50 473/473 [==============================] - 19s 41ms/step - loss: 0.6599 - accuracy: 0.7342 - val_loss: 0.5979 - val_accuracy: 0.7593 - lr: 1.0000e-04 Epoch 22/50 473/473 [==============================] - 19s 39ms/step - loss: 0.6470 - accuracy: 0.7419 - val_loss: 0.6020 - val_accuracy: 0.7637 - lr: 1.0000e-04 Epoch 23/50 473/473 [==============================] - 18s 39ms/step - loss: 0.6533 - accuracy: 0.7359 - val_loss: 0.5943 - val_accuracy: 0.7641 - lr: 1.0000e-04 Epoch 24/50 473/473 [==============================] - 18s 38ms/step - loss: 0.6441 - accuracy: 0.7425 - val_loss: 0.5866 - val_accuracy: 0.7681 - lr: 1.0000e-04 Epoch 25/50 473/473 [==============================] - 18s 39ms/step - loss: 0.6351 - accuracy: 0.7438 - val_loss: 0.5952 - val_accuracy: 0.7619 - lr: 1.0000e-04 Epoch 26/50 473/473 [==============================] - 18s 38ms/step - loss: 0.6455 - accuracy: 0.7439 - val_loss: 0.5825 - val_accuracy: 0.7689 - lr: 1.0000e-04 Epoch 27/50 473/473 [==============================] - 19s 40ms/step - loss: 0.6396 - accuracy: 0.7444 - val_loss: 0.5929 - val_accuracy: 0.7609 - lr: 1.0000e-04 Epoch 28/50 473/473 [==============================] - 19s 40ms/step - loss: 0.6401 - accuracy: 0.7401 - val_loss: 0.5803 - val_accuracy: 0.7742 - lr: 1.0000e-04 Epoch 29/50 473/473 [==============================] - 19s 41ms/step - loss: 0.6328 - accuracy: 0.7465 - val_loss: 0.5784 - val_accuracy: 0.7744 - lr: 1.0000e-04 Epoch 30/50 473/473 [==============================] - 18s 38ms/step - loss: 0.6305 - accuracy: 0.7479 - val_loss: 0.5822 - val_accuracy: 0.7715 - lr: 1.0000e-04 Epoch 31/50 473/473 [==============================] - 18s 38ms/step - loss: 0.6291 - accuracy: 0.7468 - val_loss: 0.5808 - val_accuracy: 0.7754 - lr: 1.0000e-04 Epoch 32/50 473/473 [==============================] - 18s 39ms/step - loss: 0.6255 - accuracy: 0.7497 - val_loss: 0.5841 - val_accuracy: 0.7707 - lr: 1.0000e-04 Epoch 33/50 473/473 [==============================] - 18s 38ms/step - loss: 0.6215 - accuracy: 0.7494 - val_loss: 0.5776 - val_accuracy: 0.7677 - lr: 1.0000e-04 Epoch 34/50 473/473 [==============================] - 18s 38ms/step - loss: 0.6125 - accuracy: 0.7530 - val_loss: 0.5772 - val_accuracy: 0.7738 - lr: 1.0000e-04 Epoch 35/50 473/473 [==============================] - 18s 38ms/step - loss: 0.6201 - accuracy: 0.7522 - val_loss: 0.5783 - val_accuracy: 0.7736 - lr: 1.0000e-04 Epoch 36/50 472/473 [============================>.] - ETA: 0s - loss: 0.6184 - accuracy: 0.7519Restoring model weights from the end of the best epoch: 31. 473/473 [==============================] - 18s 38ms/step - loss: 0.6186 - accuracy: 0.7520 - val_loss: 0.5763 - val_accuracy: 0.7705 - lr: 1.0000e-04 Epoch 36: early stopping
plot_model_performance_and_confusion_matrix(cnn_model_updated, history, test_generator,
metrics_function=lambda true, pred: print(classification_report(true, pred)))
from IPython.display import HTML, display
# Assuming val_loss and val_accuracy are defined from your model's evaluation
# Evaluate the model on the test set
val_loss, val_accuracy = cnn_model_updated.evaluate(
test_generator,
steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
# Format the evaluation results as HTML
output_html = f"""
<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>
<p style='color: red;'><b>Test Loss:</b> CNN_MODEL_updated </p>
<p style='color: red;'><b>Test Loss:</b> {val_loss:.4f}</p>
<p style='color: blue;'><b>Test Accuracy:</b> {val_accuracy:.4%} </p>
</div>
"""
# Display the styled output
display(HTML(output_html))
4/4 [==============================] - 0s 10ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
precision recall f1-score support
0 0.82 0.84 0.83 32
1 0.72 0.81 0.76 32
2 0.79 0.69 0.73 32
3 0.97 0.94 0.95 32
accuracy 0.82 128
macro avg 0.82 0.82 0.82 128
weighted avg 0.82 0.82 0.82 128
4/4 [==============================] - 0s 11ms/step - loss: 0.5187 - accuracy: 0.8203 Test Loss: 0.5187498927116394 Test Accuracy: 0.8203125
Test Loss: CNN_MODEL_updated
Test Loss: 0.5187
Test Accuracy: 82.0312%
Observations and Insights:__
While building the CNN models, i tried out various optimizers such as SGD, Adamax, AdamW, Adam, After running the model training several times with each and adjusting the learning rate along with the drop out
Model Architecture: The model is a deep convolutional neural network (CNN) designed with multiple convolutional blocks. Each block consists of convolutional layers with increasing filter sizes, BatchNormalization layers, MaxPooling for down-sampling, and Dropout layers to mitigate overfitting. The model concludes with dense layers and a softmax output layer for 4 classes. It uses GlobalAveragePooling2D to reduce the number of parameters and potentially decrease overfitting compared to using a Flatten layer.
Optimization and Callbacks: The model is compiled with the Adamax optimizer, a variant of Adam, with a low learning rate of 0.00018 to facilitate fine-grained updates to the model weights. Ituses callbacks such as EarlyStopping to halt training when validation accuracy stops to improve, ReduceLROnPlateau to decrease the learning rate when a metric has stopped improving, and ModelCheckpoint to save the model at its best performance based on validation accuracy.
Performance Metrics:
Accuracy: The overall accuracy on the validation set is 80%, indicating that the model correctly predicts the class for 80 out of every 100 images it is tested against. Precision, Recall, and F1-Score: The model shows varying levels of precision and recall across different classes: For the 'happy' class, it has a high precision of 0.87 but a slightly lower recall of 0.81, resulting in an F1-score of 0.84. This indicates it is quite good at identifying 'happy' faces but slightly conservative, missing some true 'happy' instances. The 'neutral' class has a precision of 0.65 with a high recall of 0.88, leading to an F1-score of 0.75. The model tends to label too many faces as 'neutral', but when it predicts 'neutral', it is often correct. For 'sad', precision is 0.81 and recall is 0.69, with an F1-score of 0.75. The model is relatively precise in identifying 'sad' faces but misses a significant portion of them. The 'surprise' class shows excellent precision at 0.96 and recall at 0.84, with an F1-score of 0.90. The model is highly effective at identifying 'surprise' faces with minimal false positives. Macro and Weighted Averages: Both macro and weighted averages across metrics hover around 0.82 for precision and 0.81 for the F1-score, suggesting a balanced performance across classes despite the variance in individual class metrics. Analysis: The model demonstrates good general performance with an overall accuracy of 80%. However, there's room for improvement, especially in balancing precision and recall across the different classes. The relatively lower precision for 'neutral' suggests a tendency to over-classify images as 'neutral', whereas the lower recall for 'sad' indicates missed detections of this emotion. The high performance on the 'surprise' class, with both high precision and recall, showcases the model's capability in identifying distinct features effectively.
Recommendations for Improvement:
Data Augmentation: To improve model generalization, especially for underperforming classes. Class Weights: Adjusting class weights during training could help address imbalances in precision and recall. Hyperparameter Tuning: Experimenting with different learning rates, optimizers, and dropout rates could enhance model performance. Model Architecture Adjustments: Tweaking the number of filters, layers, or incorporating additional regularization techniques might yield better results. Overall, cnn_model_2 shows promising results with a solid foundation for further optimization and tuning to enhance its ability to generalize across a range of facial expressions.
Observations and Insights:__ Some breif observaions, the models performed ok, reaching a highpoint of prediction accuracy of about 79 percent with cnn model 2. The precision and recall and f1 scores all are between 78 and 80 which is a good indicator. Still there is plenty of room for improvement and possibly with transferlearing the models will be able to get up to the 90 percent accuracy percent tile range.
In this section, we will create several Transfer Learning architectures. For the pre-trained models, we will select three popular architectures namely, VGG16, ResNet v2, and Efficient Net. The difference between these architectures and the previous architectures is that these will require 3 input channels while the earlier ones worked on 'grayscale' images. Therefore, we need to create new DataLoaders.
In this section, we are creating data loaders that we will use as inputs to our Neural Network. We will have to go with color_mode = 'rgb' as this is the required format for the transfer learning architectures.
# Set the random seed
np.random.seed(42)
train_generator = train_datagen.flow_from_directory(
train_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical'
)
validation_generator = test_val_datagen.flow_from_directory(
validation_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical'
)
test_generator = test_val_datagen.flow_from_directory(
test_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical',
shuffle=False # Keep data in order for evaluation
)
Found 15109 images belonging to 4 classes. Found 4977 images belonging to 4 classes. Found 128 images belonging to 4 classes.
from tensorflow.keras.applications.vgg16 import VGG16
from tensorflow.keras import Model
vgg = VGG16(include_top = False, weights = 'imagenet', input_shape = (48, 48, 3))
vgg.summary()
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/vgg16/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5
58889256/58889256 [==============================] - 2s 0us/step
Model: "vgg16"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 48, 48, 3)] 0
block1_conv1 (Conv2D) (None, 48, 48, 64) 1792
block1_conv2 (Conv2D) (None, 48, 48, 64) 36928
block1_pool (MaxPooling2D) (None, 24, 24, 64) 0
block2_conv1 (Conv2D) (None, 24, 24, 128) 73856
block2_conv2 (Conv2D) (None, 24, 24, 128) 147584
block2_pool (MaxPooling2D) (None, 12, 12, 128) 0
block3_conv1 (Conv2D) (None, 12, 12, 256) 295168
block3_conv2 (Conv2D) (None, 12, 12, 256) 590080
block3_conv3 (Conv2D) (None, 12, 12, 256) 590080
block3_pool (MaxPooling2D) (None, 6, 6, 256) 0
block4_conv1 (Conv2D) (None, 6, 6, 512) 1180160
block4_conv2 (Conv2D) (None, 6, 6, 512) 2359808
block4_conv3 (Conv2D) (None, 6, 6, 512) 2359808
block4_pool (MaxPooling2D) (None, 3, 3, 512) 0
block5_conv1 (Conv2D) (None, 3, 3, 512) 2359808
block5_conv2 (Conv2D) (None, 3, 3, 512) 2359808
block5_conv3 (Conv2D) (None, 3, 3, 512) 2359808
block5_pool (MaxPooling2D) (None, 1, 1, 512) 0
=================================================================
Total params: 14714688 (56.13 MB)
Trainable params: 14714688 (56.13 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
np.random.seed(42)
samples_per_class = {
'surprise': 3173,
'happy': 3976,
'neutral': 3978,
'sad': 3982,
}
# Calculate total samples
total_samples = sum(samples_per_class.values())
# Calculate class weights
class_weights = {class_id: total_samples/(len(samples_per_class)*num_samples)
for class_id, num_samples in enumerate(samples_per_class.values())}
print(class_weights)
{0: 1.1904349196344153, 1: 0.9500125754527163, 2: 0.949534942182001, 3: 0.9485811150175791}
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=False,
fill_mode='nearest'
)
# Define the image size and paths
image_size = 48
train_path = "Facial_emotion_images/train"
test_path = "Facial_emotion_images/test"
validation_path = "Facial_emotion_images/validation"
# Initialize the data generator with rescaling
datagen = ImageDataGenerator(rescale=1./255)
# Prepare the data generators
train_generator = datagen.flow_from_directory(
train_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical')
validation_generator = datagen.flow_from_directory(
validation_path,
target_size=(48, 48),
batch_size=32, # Keep the same batch size for simplicity
class_mode='categorical')
test_generator = datagen.flow_from_directory(
test_path,
target_size=(48, 48),
shuffle=False)
Found 15109 images belonging to 4 classes. Found 4977 images belonging to 4 classes. Found 128 images belonging to 4 classes.
# Set the random seed
np.random.seed(42)
from tensorflow.keras.applications.vgg16 import VGG16
from tensorflow.keras.layers import Dense, Flatten, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import AdamW, SGD, Adamax
# Set the random seed
np.random.seed(42)
#
# Load the VGG16 model
vgg = VGG16(include_top=False, weights='imagenet', input_shape=(48, 48, 3))
# Freeze all layers in the base VGG16 model
for layer in vgg.layers:
layer.trainable = False
# Select the output of the 'block3_pool' layer
x = vgg.get_layer('block3_pool').output
# Continue adding your custom layers from here
x = Flatten()(x) # Flatten the output of 'block3_pool'
x = Dense(512, activation='relu')(x) # Custom dense layer
x = Dropout(0.1)(x) # Dropout layer
x = Dense(1024, activation='relu')(x) # Another dense layer
x = Dropout(0.2)(x) # Dropout layer
x = Dense(32, activation='relu')(x)
# Final layer with softmax for classification
predictions = Dense(4, activation='softmax')(x) # Assuming 4 classes for emotions
# Create the final model using VGG16 input and your custom output
model = Model(inputs=vgg.input, outputs=predictions)
# Compile the model
model.compile(optimizer=Adamax(learning_rate=0.000025), loss='categorical_crossentropy', metrics=['accuracy'])
model.summary()
Model: "model_3"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_5 (InputLayer) [(None, 48, 48, 3)] 0
block1_conv1 (Conv2D) (None, 48, 48, 64) 1792
block1_conv2 (Conv2D) (None, 48, 48, 64) 36928
block1_pool (MaxPooling2D) (None, 24, 24, 64) 0
block2_conv1 (Conv2D) (None, 24, 24, 128) 73856
block2_conv2 (Conv2D) (None, 24, 24, 128) 147584
block2_pool (MaxPooling2D) (None, 12, 12, 128) 0
block3_conv1 (Conv2D) (None, 12, 12, 256) 295168
block3_conv2 (Conv2D) (None, 12, 12, 256) 590080
block3_conv3 (Conv2D) (None, 12, 12, 256) 590080
block3_pool (MaxPooling2D) (None, 6, 6, 256) 0
flatten_7 (Flatten) (None, 9216) 0
dense_42 (Dense) (None, 512) 4719104
dropout_55 (Dropout) (None, 512) 0
dense_43 (Dense) (None, 1024) 525312
dropout_56 (Dropout) (None, 1024) 0
dense_44 (Dense) (None, 32) 32800
dense_45 (Dense) (None, 4) 132
=================================================================
Total params: 7012836 (26.75 MB)
Trainable params: 5277348 (20.13 MB)
Non-trainable params: 1735488 (6.62 MB)
_________________________________________________________________
from tensorflow.keras.callbacks import ReduceLROnPlateau, ModelCheckpoint, EarlyStopping
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, min_lr=0.000001)
model_checkpoint = ModelCheckpoint('best_model.h5', monitor='val_accuracy', save_best_only=True)
early_stopping = EarlyStopping(monitor='val_loss', patience=7, restore_best_weights=True)
# Train the model
history = model.fit(
train_generator,
epochs=75,
validation_data=validation_generator,
callbacks=[reduce_lr, model_checkpoint, early_stopping]
)
Epoch 1/75 473/473 [==============================] - 10s 17ms/step - loss: 1.8805 - accuracy: 0.2998 - val_loss: 1.3816 - val_accuracy: 0.2620 - lr: 2.5000e-05 Epoch 2/75 473/473 [==============================] - 8s 17ms/step - loss: 1.3644 - accuracy: 0.3341 - val_loss: 1.3035 - val_accuracy: 0.3709 - lr: 2.5000e-05 Epoch 3/75 473/473 [==============================] - 8s 16ms/step - loss: 1.2823 - accuracy: 0.4023 - val_loss: 1.2123 - val_accuracy: 0.4571 - lr: 2.5000e-05 Epoch 4/75 473/473 [==============================] - 8s 16ms/step - loss: 1.2114 - accuracy: 0.4331 - val_loss: 1.1272 - val_accuracy: 0.5120 - lr: 2.5000e-05 Epoch 5/75 473/473 [==============================] - 8s 16ms/step - loss: 1.1606 - accuracy: 0.4671 - val_loss: 1.1128 - val_accuracy: 0.5310 - lr: 2.5000e-05 Epoch 6/75 473/473 [==============================] - 8s 16ms/step - loss: 1.1103 - accuracy: 0.4885 - val_loss: 1.0563 - val_accuracy: 0.5429 - lr: 2.5000e-05 Epoch 7/75 473/473 [==============================] - 7s 16ms/step - loss: 1.0590 - accuracy: 0.5169 - val_loss: 0.9952 - val_accuracy: 0.5626 - lr: 2.5000e-05 Epoch 8/75 473/473 [==============================] - 8s 16ms/step - loss: 1.0040 - accuracy: 0.5544 - val_loss: 0.9309 - val_accuracy: 0.6168 - lr: 2.5000e-05 Epoch 9/75 473/473 [==============================] - 8s 16ms/step - loss: 0.9363 - accuracy: 0.5885 - val_loss: 0.8933 - val_accuracy: 0.6359 - lr: 2.5000e-05 Epoch 10/75 473/473 [==============================] - 8s 16ms/step - loss: 0.8996 - accuracy: 0.6125 - val_loss: 0.8726 - val_accuracy: 0.6418 - lr: 2.5000e-05 Epoch 11/75 473/473 [==============================] - 8s 16ms/step - loss: 0.8622 - accuracy: 0.6273 - val_loss: 0.8536 - val_accuracy: 0.6546 - lr: 2.5000e-05 Epoch 12/75 473/473 [==============================] - 8s 16ms/step - loss: 0.8274 - accuracy: 0.6489 - val_loss: 0.8336 - val_accuracy: 0.6612 - lr: 2.5000e-05 Epoch 13/75 473/473 [==============================] - 8s 17ms/step - loss: 0.8026 - accuracy: 0.6589 - val_loss: 0.8221 - val_accuracy: 0.6620 - lr: 2.5000e-05 Epoch 14/75 473/473 [==============================] - 8s 16ms/step - loss: 0.7751 - accuracy: 0.6736 - val_loss: 0.8141 - val_accuracy: 0.6775 - lr: 2.5000e-05 Epoch 15/75 473/473 [==============================] - 7s 16ms/step - loss: 0.7490 - accuracy: 0.6816 - val_loss: 0.8283 - val_accuracy: 0.6739 - lr: 2.5000e-05 Epoch 16/75 473/473 [==============================] - 8s 16ms/step - loss: 0.7240 - accuracy: 0.6998 - val_loss: 0.7986 - val_accuracy: 0.6801 - lr: 2.5000e-05 Epoch 17/75 473/473 [==============================] - 8s 16ms/step - loss: 0.6963 - accuracy: 0.7090 - val_loss: 0.7856 - val_accuracy: 0.6856 - lr: 2.5000e-05 Epoch 18/75 473/473 [==============================] - 7s 16ms/step - loss: 0.6791 - accuracy: 0.7133 - val_loss: 0.7866 - val_accuracy: 0.6845 - lr: 2.5000e-05 Epoch 19/75 473/473 [==============================] - 8s 17ms/step - loss: 0.6501 - accuracy: 0.7321 - val_loss: 0.7696 - val_accuracy: 0.6922 - lr: 2.5000e-05 Epoch 20/75 473/473 [==============================] - 7s 16ms/step - loss: 0.6265 - accuracy: 0.7390 - val_loss: 0.8019 - val_accuracy: 0.6894 - lr: 2.5000e-05 Epoch 21/75 473/473 [==============================] - 8s 16ms/step - loss: 0.6100 - accuracy: 0.7475 - val_loss: 0.7730 - val_accuracy: 0.6940 - lr: 2.5000e-05 Epoch 22/75 473/473 [==============================] - 8s 16ms/step - loss: 0.5893 - accuracy: 0.7551 - val_loss: 0.7784 - val_accuracy: 0.6920 - lr: 2.5000e-05 Epoch 23/75 473/473 [==============================] - 8s 16ms/step - loss: 0.5662 - accuracy: 0.7686 - val_loss: 0.7709 - val_accuracy: 0.6994 - lr: 2.5000e-05 Epoch 24/75 473/473 [==============================] - 7s 16ms/step - loss: 0.5509 - accuracy: 0.7749 - val_loss: 0.7765 - val_accuracy: 0.6946 - lr: 2.5000e-05 Epoch 25/75 473/473 [==============================] - 8s 16ms/step - loss: 0.5189 - accuracy: 0.7917 - val_loss: 0.7767 - val_accuracy: 0.6966 - lr: 5.0000e-06 Epoch 26/75 473/473 [==============================] - 8s 16ms/step - loss: 0.5130 - accuracy: 0.7920 - val_loss: 0.7657 - val_accuracy: 0.7024 - lr: 5.0000e-06 Epoch 27/75 473/473 [==============================] - 8s 16ms/step - loss: 0.5047 - accuracy: 0.7953 - val_loss: 0.7609 - val_accuracy: 0.7018 - lr: 5.0000e-06 Epoch 28/75 473/473 [==============================] - 8s 16ms/step - loss: 0.4956 - accuracy: 0.8018 - val_loss: 0.7666 - val_accuracy: 0.7036 - lr: 5.0000e-06 Epoch 29/75 473/473 [==============================] - 7s 16ms/step - loss: 0.4924 - accuracy: 0.7997 - val_loss: 0.7728 - val_accuracy: 0.6998 - lr: 5.0000e-06 Epoch 30/75 473/473 [==============================] - 7s 16ms/step - loss: 0.4924 - accuracy: 0.8024 - val_loss: 0.7701 - val_accuracy: 0.7028 - lr: 5.0000e-06 Epoch 31/75 473/473 [==============================] - 8s 16ms/step - loss: 0.4835 - accuracy: 0.8064 - val_loss: 0.7681 - val_accuracy: 0.7018 - lr: 5.0000e-06 Epoch 32/75 473/473 [==============================] - 7s 16ms/step - loss: 0.4756 - accuracy: 0.8108 - val_loss: 0.7696 - val_accuracy: 0.7012 - lr: 5.0000e-06 Epoch 33/75 473/473 [==============================] - 8s 16ms/step - loss: 0.4751 - accuracy: 0.8079 - val_loss: 0.7710 - val_accuracy: 0.7026 - lr: 1.0000e-06 Epoch 34/75 473/473 [==============================] - 7s 15ms/step - loss: 0.4773 - accuracy: 0.8082 - val_loss: 0.7680 - val_accuracy: 0.7032 - lr: 1.0000e-06
from sklearn.metrics import confusion_matrix, classification_report
cm = confusion_matrix(test_generator.classes, np.argmax(model.predict(test_generator), axis=1))
# Confusion matrix
print("Confusion Matrix:")
print(cm)
# Classification report
# Plot the training and validation accuracy values
plt.figure(figsize=(15, 5))
plt.subplot(1, 3, 1)
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('VGG_Model_1 Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Plot the training and validation loss values
plt.subplot(1, 3, 2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('VGG_Model_1 Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Plot the training and validation loss values
plt.subplot(1, 3, 3)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", cbar=False)
plt.xlabel("Predicted Label")
plt.ylabel("True Label")
plt.title("Confusion Matrix for VGG_MODEL_1")
plt.show()
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix
predicted_labels = np.argmax(model.predict(test_generator), axis=1)
# Get the true labels for the test set
true_labels = test_generator.classes
# Get the class names
class_names_list = test_generator.class_indices.keys()
print(class_names_list)
# Calculate and display the metrics
metrics_score(true_labels, predicted_labels)
# Plot the confusion matrix
cm = confusion_matrix(true_labels, predicted_labels)
4/4 [==============================] - 0s 11ms/step Confusion Matrix: [[25 6 0 1] [ 0 28 3 1] [ 3 8 18 3] [ 0 2 2 28]]
4/4 [==============================] - 0s 13ms/step
dict_keys(['happy', 'neutral', 'sad', 'surprise'])
precision recall f1-score support
0 0.89 0.78 0.83 32
1 0.64 0.88 0.74 32
2 0.78 0.56 0.65 32
3 0.85 0.88 0.86 32
accuracy 0.77 128
macro avg 0.79 0.77 0.77 128
weighted avg 0.79 0.77 0.77 128
The customized VGG16 model, adapted for emotion classification into four categories (happy, neutral, sad, surprise), demonstrates an overall accuracy of 77%. The model shows varying precision and recall across classes, with notable strengths in identifying 'surprise' emotions accurately. Despite its solid foundation, there's room for improvement, especially in balancing performance across all classes. Recommendations include data augmentation, fine-tuning of VGG16 layers, optimization of hyperparameters, and adjustment of class weights to enhance generalization and performance uniformly across categories.
Think About It:
**Observations and Insights:__It is seeming to prove to be difficult to get the model performance above 75-80 percent accuracy. Additionally, The Category of neutral is often what gets predicted when the actual class is that of 'sad', The largest confusion point seems to be a sad face being miss classified for 'neutral'
# Set the random Seed
np.random.seed(42)
from tensorflow.keras.applications.resnet50 import ResNet50, preprocess_input
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, Flatten, Dropout
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau
import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.applications.resnet50 import preprocess_input, ResNet50
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D
import tensorflow as tf
import tensorflow.keras.applications as ap
from tensorflow.keras import Model
Resnet = ap.ResNet101(include_top = False, weights = "imagenet", input_shape=(48, 48, 3))
Resnet.summary()
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/resnet/resnet101_weights_tf_dim_ordering_tf_kernels_notop.h5
171446536/171446536 [==============================] - 6s 0us/step
Model: "resnet101"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) [(None, 48, 48, 3)] 0 []
conv1_pad (ZeroPadding2D) (None, 54, 54, 3) 0 ['input_1[0][0]']
conv1_conv (Conv2D) (None, 24, 24, 64) 9472 ['conv1_pad[0][0]']
conv1_bn (BatchNormalizati (None, 24, 24, 64) 256 ['conv1_conv[0][0]']
on)
conv1_relu (Activation) (None, 24, 24, 64) 0 ['conv1_bn[0][0]']
pool1_pad (ZeroPadding2D) (None, 26, 26, 64) 0 ['conv1_relu[0][0]']
pool1_pool (MaxPooling2D) (None, 12, 12, 64) 0 ['pool1_pad[0][0]']
conv2_block1_1_conv (Conv2 (None, 12, 12, 64) 4160 ['pool1_pool[0][0]']
D)
conv2_block1_1_bn (BatchNo (None, 12, 12, 64) 256 ['conv2_block1_1_conv[0][0]']
rmalization)
conv2_block1_1_relu (Activ (None, 12, 12, 64) 0 ['conv2_block1_1_bn[0][0]']
ation)
conv2_block1_2_conv (Conv2 (None, 12, 12, 64) 36928 ['conv2_block1_1_relu[0][0]']
D)
conv2_block1_2_bn (BatchNo (None, 12, 12, 64) 256 ['conv2_block1_2_conv[0][0]']
rmalization)
conv2_block1_2_relu (Activ (None, 12, 12, 64) 0 ['conv2_block1_2_bn[0][0]']
ation)
conv2_block1_0_conv (Conv2 (None, 12, 12, 256) 16640 ['pool1_pool[0][0]']
D)
conv2_block1_3_conv (Conv2 (None, 12, 12, 256) 16640 ['conv2_block1_2_relu[0][0]']
D)
conv2_block1_0_bn (BatchNo (None, 12, 12, 256) 1024 ['conv2_block1_0_conv[0][0]']
rmalization)
conv2_block1_3_bn (BatchNo (None, 12, 12, 256) 1024 ['conv2_block1_3_conv[0][0]']
rmalization)
conv2_block1_add (Add) (None, 12, 12, 256) 0 ['conv2_block1_0_bn[0][0]',
'conv2_block1_3_bn[0][0]']
conv2_block1_out (Activati (None, 12, 12, 256) 0 ['conv2_block1_add[0][0]']
on)
conv2_block2_1_conv (Conv2 (None, 12, 12, 64) 16448 ['conv2_block1_out[0][0]']
D)
conv2_block2_1_bn (BatchNo (None, 12, 12, 64) 256 ['conv2_block2_1_conv[0][0]']
rmalization)
conv2_block2_1_relu (Activ (None, 12, 12, 64) 0 ['conv2_block2_1_bn[0][0]']
ation)
conv2_block2_2_conv (Conv2 (None, 12, 12, 64) 36928 ['conv2_block2_1_relu[0][0]']
D)
conv2_block2_2_bn (BatchNo (None, 12, 12, 64) 256 ['conv2_block2_2_conv[0][0]']
rmalization)
conv2_block2_2_relu (Activ (None, 12, 12, 64) 0 ['conv2_block2_2_bn[0][0]']
ation)
conv2_block2_3_conv (Conv2 (None, 12, 12, 256) 16640 ['conv2_block2_2_relu[0][0]']
D)
conv2_block2_3_bn (BatchNo (None, 12, 12, 256) 1024 ['conv2_block2_3_conv[0][0]']
rmalization)
conv2_block2_add (Add) (None, 12, 12, 256) 0 ['conv2_block1_out[0][0]',
'conv2_block2_3_bn[0][0]']
conv2_block2_out (Activati (None, 12, 12, 256) 0 ['conv2_block2_add[0][0]']
on)
conv2_block3_1_conv (Conv2 (None, 12, 12, 64) 16448 ['conv2_block2_out[0][0]']
D)
conv2_block3_1_bn (BatchNo (None, 12, 12, 64) 256 ['conv2_block3_1_conv[0][0]']
rmalization)
conv2_block3_1_relu (Activ (None, 12, 12, 64) 0 ['conv2_block3_1_bn[0][0]']
ation)
conv2_block3_2_conv (Conv2 (None, 12, 12, 64) 36928 ['conv2_block3_1_relu[0][0]']
D)
conv2_block3_2_bn (BatchNo (None, 12, 12, 64) 256 ['conv2_block3_2_conv[0][0]']
rmalization)
conv2_block3_2_relu (Activ (None, 12, 12, 64) 0 ['conv2_block3_2_bn[0][0]']
ation)
conv2_block3_3_conv (Conv2 (None, 12, 12, 256) 16640 ['conv2_block3_2_relu[0][0]']
D)
conv2_block3_3_bn (BatchNo (None, 12, 12, 256) 1024 ['conv2_block3_3_conv[0][0]']
rmalization)
conv2_block3_add (Add) (None, 12, 12, 256) 0 ['conv2_block2_out[0][0]',
'conv2_block3_3_bn[0][0]']
conv2_block3_out (Activati (None, 12, 12, 256) 0 ['conv2_block3_add[0][0]']
on)
conv3_block1_1_conv (Conv2 (None, 6, 6, 128) 32896 ['conv2_block3_out[0][0]']
D)
conv3_block1_1_bn (BatchNo (None, 6, 6, 128) 512 ['conv3_block1_1_conv[0][0]']
rmalization)
conv3_block1_1_relu (Activ (None, 6, 6, 128) 0 ['conv3_block1_1_bn[0][0]']
ation)
conv3_block1_2_conv (Conv2 (None, 6, 6, 128) 147584 ['conv3_block1_1_relu[0][0]']
D)
conv3_block1_2_bn (BatchNo (None, 6, 6, 128) 512 ['conv3_block1_2_conv[0][0]']
rmalization)
conv3_block1_2_relu (Activ (None, 6, 6, 128) 0 ['conv3_block1_2_bn[0][0]']
ation)
conv3_block1_0_conv (Conv2 (None, 6, 6, 512) 131584 ['conv2_block3_out[0][0]']
D)
conv3_block1_3_conv (Conv2 (None, 6, 6, 512) 66048 ['conv3_block1_2_relu[0][0]']
D)
conv3_block1_0_bn (BatchNo (None, 6, 6, 512) 2048 ['conv3_block1_0_conv[0][0]']
rmalization)
conv3_block1_3_bn (BatchNo (None, 6, 6, 512) 2048 ['conv3_block1_3_conv[0][0]']
rmalization)
conv3_block1_add (Add) (None, 6, 6, 512) 0 ['conv3_block1_0_bn[0][0]',
'conv3_block1_3_bn[0][0]']
conv3_block1_out (Activati (None, 6, 6, 512) 0 ['conv3_block1_add[0][0]']
on)
conv3_block2_1_conv (Conv2 (None, 6, 6, 128) 65664 ['conv3_block1_out[0][0]']
D)
conv3_block2_1_bn (BatchNo (None, 6, 6, 128) 512 ['conv3_block2_1_conv[0][0]']
rmalization)
conv3_block2_1_relu (Activ (None, 6, 6, 128) 0 ['conv3_block2_1_bn[0][0]']
ation)
conv3_block2_2_conv (Conv2 (None, 6, 6, 128) 147584 ['conv3_block2_1_relu[0][0]']
D)
conv3_block2_2_bn (BatchNo (None, 6, 6, 128) 512 ['conv3_block2_2_conv[0][0]']
rmalization)
conv3_block2_2_relu (Activ (None, 6, 6, 128) 0 ['conv3_block2_2_bn[0][0]']
ation)
conv3_block2_3_conv (Conv2 (None, 6, 6, 512) 66048 ['conv3_block2_2_relu[0][0]']
D)
conv3_block2_3_bn (BatchNo (None, 6, 6, 512) 2048 ['conv3_block2_3_conv[0][0]']
rmalization)
conv3_block2_add (Add) (None, 6, 6, 512) 0 ['conv3_block1_out[0][0]',
'conv3_block2_3_bn[0][0]']
conv3_block2_out (Activati (None, 6, 6, 512) 0 ['conv3_block2_add[0][0]']
on)
conv3_block3_1_conv (Conv2 (None, 6, 6, 128) 65664 ['conv3_block2_out[0][0]']
D)
conv3_block3_1_bn (BatchNo (None, 6, 6, 128) 512 ['conv3_block3_1_conv[0][0]']
rmalization)
conv3_block3_1_relu (Activ (None, 6, 6, 128) 0 ['conv3_block3_1_bn[0][0]']
ation)
conv3_block3_2_conv (Conv2 (None, 6, 6, 128) 147584 ['conv3_block3_1_relu[0][0]']
D)
conv3_block3_2_bn (BatchNo (None, 6, 6, 128) 512 ['conv3_block3_2_conv[0][0]']
rmalization)
conv3_block3_2_relu (Activ (None, 6, 6, 128) 0 ['conv3_block3_2_bn[0][0]']
ation)
conv3_block3_3_conv (Conv2 (None, 6, 6, 512) 66048 ['conv3_block3_2_relu[0][0]']
D)
conv3_block3_3_bn (BatchNo (None, 6, 6, 512) 2048 ['conv3_block3_3_conv[0][0]']
rmalization)
conv3_block3_add (Add) (None, 6, 6, 512) 0 ['conv3_block2_out[0][0]',
'conv3_block3_3_bn[0][0]']
conv3_block3_out (Activati (None, 6, 6, 512) 0 ['conv3_block3_add[0][0]']
on)
conv3_block4_1_conv (Conv2 (None, 6, 6, 128) 65664 ['conv3_block3_out[0][0]']
D)
conv3_block4_1_bn (BatchNo (None, 6, 6, 128) 512 ['conv3_block4_1_conv[0][0]']
rmalization)
conv3_block4_1_relu (Activ (None, 6, 6, 128) 0 ['conv3_block4_1_bn[0][0]']
ation)
conv3_block4_2_conv (Conv2 (None, 6, 6, 128) 147584 ['conv3_block4_1_relu[0][0]']
D)
conv3_block4_2_bn (BatchNo (None, 6, 6, 128) 512 ['conv3_block4_2_conv[0][0]']
rmalization)
conv3_block4_2_relu (Activ (None, 6, 6, 128) 0 ['conv3_block4_2_bn[0][0]']
ation)
conv3_block4_3_conv (Conv2 (None, 6, 6, 512) 66048 ['conv3_block4_2_relu[0][0]']
D)
conv3_block4_3_bn (BatchNo (None, 6, 6, 512) 2048 ['conv3_block4_3_conv[0][0]']
rmalization)
conv3_block4_add (Add) (None, 6, 6, 512) 0 ['conv3_block3_out[0][0]',
'conv3_block4_3_bn[0][0]']
conv3_block4_out (Activati (None, 6, 6, 512) 0 ['conv3_block4_add[0][0]']
on)
conv4_block1_1_conv (Conv2 (None, 3, 3, 256) 131328 ['conv3_block4_out[0][0]']
D)
conv4_block1_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block1_1_conv[0][0]']
rmalization)
conv4_block1_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block1_1_bn[0][0]']
ation)
conv4_block1_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block1_1_relu[0][0]']
D)
conv4_block1_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block1_2_conv[0][0]']
rmalization)
conv4_block1_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block1_2_bn[0][0]']
ation)
conv4_block1_0_conv (Conv2 (None, 3, 3, 1024) 525312 ['conv3_block4_out[0][0]']
D)
conv4_block1_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block1_2_relu[0][0]']
D)
conv4_block1_0_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block1_0_conv[0][0]']
rmalization)
conv4_block1_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block1_3_conv[0][0]']
rmalization)
conv4_block1_add (Add) (None, 3, 3, 1024) 0 ['conv4_block1_0_bn[0][0]',
'conv4_block1_3_bn[0][0]']
conv4_block1_out (Activati (None, 3, 3, 1024) 0 ['conv4_block1_add[0][0]']
on)
conv4_block2_1_conv (Conv2 (None, 3, 3, 256) 262400 ['conv4_block1_out[0][0]']
D)
conv4_block2_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block2_1_conv[0][0]']
rmalization)
conv4_block2_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block2_1_bn[0][0]']
ation)
conv4_block2_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block2_1_relu[0][0]']
D)
conv4_block2_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block2_2_conv[0][0]']
rmalization)
conv4_block2_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block2_2_bn[0][0]']
ation)
conv4_block2_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block2_2_relu[0][0]']
D)
conv4_block2_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block2_3_conv[0][0]']
rmalization)
conv4_block2_add (Add) (None, 3, 3, 1024) 0 ['conv4_block1_out[0][0]',
'conv4_block2_3_bn[0][0]']
conv4_block2_out (Activati (None, 3, 3, 1024) 0 ['conv4_block2_add[0][0]']
on)
conv4_block3_1_conv (Conv2 (None, 3, 3, 256) 262400 ['conv4_block2_out[0][0]']
D)
conv4_block3_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block3_1_conv[0][0]']
rmalization)
conv4_block3_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block3_1_bn[0][0]']
ation)
conv4_block3_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block3_1_relu[0][0]']
D)
conv4_block3_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block3_2_conv[0][0]']
rmalization)
conv4_block3_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block3_2_bn[0][0]']
ation)
conv4_block3_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block3_2_relu[0][0]']
D)
conv4_block3_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block3_3_conv[0][0]']
rmalization)
conv4_block3_add (Add) (None, 3, 3, 1024) 0 ['conv4_block2_out[0][0]',
'conv4_block3_3_bn[0][0]']
conv4_block3_out (Activati (None, 3, 3, 1024) 0 ['conv4_block3_add[0][0]']
on)
conv4_block4_1_conv (Conv2 (None, 3, 3, 256) 262400 ['conv4_block3_out[0][0]']
D)
conv4_block4_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block4_1_conv[0][0]']
rmalization)
conv4_block4_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block4_1_bn[0][0]']
ation)
conv4_block4_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block4_1_relu[0][0]']
D)
conv4_block4_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block4_2_conv[0][0]']
rmalization)
conv4_block4_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block4_2_bn[0][0]']
ation)
conv4_block4_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block4_2_relu[0][0]']
D)
conv4_block4_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block4_3_conv[0][0]']
rmalization)
conv4_block4_add (Add) (None, 3, 3, 1024) 0 ['conv4_block3_out[0][0]',
'conv4_block4_3_bn[0][0]']
conv4_block4_out (Activati (None, 3, 3, 1024) 0 ['conv4_block4_add[0][0]']
on)
conv4_block5_1_conv (Conv2 (None, 3, 3, 256) 262400 ['conv4_block4_out[0][0]']
D)
conv4_block5_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block5_1_conv[0][0]']
rmalization)
conv4_block5_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block5_1_bn[0][0]']
ation)
conv4_block5_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block5_1_relu[0][0]']
D)
conv4_block5_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block5_2_conv[0][0]']
rmalization)
conv4_block5_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block5_2_bn[0][0]']
ation)
conv4_block5_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block5_2_relu[0][0]']
D)
conv4_block5_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block5_3_conv[0][0]']
rmalization)
conv4_block5_add (Add) (None, 3, 3, 1024) 0 ['conv4_block4_out[0][0]',
'conv4_block5_3_bn[0][0]']
conv4_block5_out (Activati (None, 3, 3, 1024) 0 ['conv4_block5_add[0][0]']
on)
conv4_block6_1_conv (Conv2 (None, 3, 3, 256) 262400 ['conv4_block5_out[0][0]']
D)
conv4_block6_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block6_1_conv[0][0]']
rmalization)
conv4_block6_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block6_1_bn[0][0]']
ation)
conv4_block6_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block6_1_relu[0][0]']
D)
conv4_block6_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block6_2_conv[0][0]']
rmalization)
conv4_block6_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block6_2_bn[0][0]']
ation)
conv4_block6_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block6_2_relu[0][0]']
D)
conv4_block6_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block6_3_conv[0][0]']
rmalization)
conv4_block6_add (Add) (None, 3, 3, 1024) 0 ['conv4_block5_out[0][0]',
'conv4_block6_3_bn[0][0]']
conv4_block6_out (Activati (None, 3, 3, 1024) 0 ['conv4_block6_add[0][0]']
on)
conv4_block7_1_conv (Conv2 (None, 3, 3, 256) 262400 ['conv4_block6_out[0][0]']
D)
conv4_block7_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block7_1_conv[0][0]']
rmalization)
conv4_block7_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block7_1_bn[0][0]']
ation)
conv4_block7_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block7_1_relu[0][0]']
D)
conv4_block7_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block7_2_conv[0][0]']
rmalization)
conv4_block7_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block7_2_bn[0][0]']
ation)
conv4_block7_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block7_2_relu[0][0]']
D)
conv4_block7_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block7_3_conv[0][0]']
rmalization)
conv4_block7_add (Add) (None, 3, 3, 1024) 0 ['conv4_block6_out[0][0]',
'conv4_block7_3_bn[0][0]']
conv4_block7_out (Activati (None, 3, 3, 1024) 0 ['conv4_block7_add[0][0]']
on)
conv4_block8_1_conv (Conv2 (None, 3, 3, 256) 262400 ['conv4_block7_out[0][0]']
D)
conv4_block8_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block8_1_conv[0][0]']
rmalization)
conv4_block8_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block8_1_bn[0][0]']
ation)
conv4_block8_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block8_1_relu[0][0]']
D)
conv4_block8_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block8_2_conv[0][0]']
rmalization)
conv4_block8_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block8_2_bn[0][0]']
ation)
conv4_block8_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block8_2_relu[0][0]']
D)
conv4_block8_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block8_3_conv[0][0]']
rmalization)
conv4_block8_add (Add) (None, 3, 3, 1024) 0 ['conv4_block7_out[0][0]',
'conv4_block8_3_bn[0][0]']
conv4_block8_out (Activati (None, 3, 3, 1024) 0 ['conv4_block8_add[0][0]']
on)
conv4_block9_1_conv (Conv2 (None, 3, 3, 256) 262400 ['conv4_block8_out[0][0]']
D)
conv4_block9_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block9_1_conv[0][0]']
rmalization)
conv4_block9_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block9_1_bn[0][0]']
ation)
conv4_block9_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block9_1_relu[0][0]']
D)
conv4_block9_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block9_2_conv[0][0]']
rmalization)
conv4_block9_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block9_2_bn[0][0]']
ation)
conv4_block9_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block9_2_relu[0][0]']
D)
conv4_block9_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block9_3_conv[0][0]']
rmalization)
conv4_block9_add (Add) (None, 3, 3, 1024) 0 ['conv4_block8_out[0][0]',
'conv4_block9_3_bn[0][0]']
conv4_block9_out (Activati (None, 3, 3, 1024) 0 ['conv4_block9_add[0][0]']
on)
conv4_block10_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block9_out[0][0]']
2D)
conv4_block10_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block10_1_conv[0][0]']
ormalization)
conv4_block10_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block10_1_bn[0][0]']
vation)
conv4_block10_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block10_1_relu[0][0]']
2D)
conv4_block10_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block10_2_conv[0][0]']
ormalization)
conv4_block10_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block10_2_bn[0][0]']
vation)
conv4_block10_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block10_2_relu[0][0]']
2D)
conv4_block10_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block10_3_conv[0][0]']
ormalization)
conv4_block10_add (Add) (None, 3, 3, 1024) 0 ['conv4_block9_out[0][0]',
'conv4_block10_3_bn[0][0]']
conv4_block10_out (Activat (None, 3, 3, 1024) 0 ['conv4_block10_add[0][0]']
ion)
conv4_block11_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block10_out[0][0]']
2D)
conv4_block11_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block11_1_conv[0][0]']
ormalization)
conv4_block11_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block11_1_bn[0][0]']
vation)
conv4_block11_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block11_1_relu[0][0]']
2D)
conv4_block11_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block11_2_conv[0][0]']
ormalization)
conv4_block11_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block11_2_bn[0][0]']
vation)
conv4_block11_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block11_2_relu[0][0]']
2D)
conv4_block11_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block11_3_conv[0][0]']
ormalization)
conv4_block11_add (Add) (None, 3, 3, 1024) 0 ['conv4_block10_out[0][0]',
'conv4_block11_3_bn[0][0]']
conv4_block11_out (Activat (None, 3, 3, 1024) 0 ['conv4_block11_add[0][0]']
ion)
conv4_block12_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block11_out[0][0]']
2D)
conv4_block12_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block12_1_conv[0][0]']
ormalization)
conv4_block12_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block12_1_bn[0][0]']
vation)
conv4_block12_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block12_1_relu[0][0]']
2D)
conv4_block12_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block12_2_conv[0][0]']
ormalization)
conv4_block12_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block12_2_bn[0][0]']
vation)
conv4_block12_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block12_2_relu[0][0]']
2D)
conv4_block12_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block12_3_conv[0][0]']
ormalization)
conv4_block12_add (Add) (None, 3, 3, 1024) 0 ['conv4_block11_out[0][0]',
'conv4_block12_3_bn[0][0]']
conv4_block12_out (Activat (None, 3, 3, 1024) 0 ['conv4_block12_add[0][0]']
ion)
conv4_block13_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block12_out[0][0]']
2D)
conv4_block13_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block13_1_conv[0][0]']
ormalization)
conv4_block13_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block13_1_bn[0][0]']
vation)
conv4_block13_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block13_1_relu[0][0]']
2D)
conv4_block13_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block13_2_conv[0][0]']
ormalization)
conv4_block13_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block13_2_bn[0][0]']
vation)
conv4_block13_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block13_2_relu[0][0]']
2D)
conv4_block13_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block13_3_conv[0][0]']
ormalization)
conv4_block13_add (Add) (None, 3, 3, 1024) 0 ['conv4_block12_out[0][0]',
'conv4_block13_3_bn[0][0]']
conv4_block13_out (Activat (None, 3, 3, 1024) 0 ['conv4_block13_add[0][0]']
ion)
conv4_block14_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block13_out[0][0]']
2D)
conv4_block14_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block14_1_conv[0][0]']
ormalization)
conv4_block14_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block14_1_bn[0][0]']
vation)
conv4_block14_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block14_1_relu[0][0]']
2D)
conv4_block14_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block14_2_conv[0][0]']
ormalization)
conv4_block14_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block14_2_bn[0][0]']
vation)
conv4_block14_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block14_2_relu[0][0]']
2D)
conv4_block14_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block14_3_conv[0][0]']
ormalization)
conv4_block14_add (Add) (None, 3, 3, 1024) 0 ['conv4_block13_out[0][0]',
'conv4_block14_3_bn[0][0]']
conv4_block14_out (Activat (None, 3, 3, 1024) 0 ['conv4_block14_add[0][0]']
ion)
conv4_block15_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block14_out[0][0]']
2D)
conv4_block15_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block15_1_conv[0][0]']
ormalization)
conv4_block15_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block15_1_bn[0][0]']
vation)
conv4_block15_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block15_1_relu[0][0]']
2D)
conv4_block15_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block15_2_conv[0][0]']
ormalization)
conv4_block15_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block15_2_bn[0][0]']
vation)
conv4_block15_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block15_2_relu[0][0]']
2D)
conv4_block15_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block15_3_conv[0][0]']
ormalization)
conv4_block15_add (Add) (None, 3, 3, 1024) 0 ['conv4_block14_out[0][0]',
'conv4_block15_3_bn[0][0]']
conv4_block15_out (Activat (None, 3, 3, 1024) 0 ['conv4_block15_add[0][0]']
ion)
conv4_block16_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block15_out[0][0]']
2D)
conv4_block16_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block16_1_conv[0][0]']
ormalization)
conv4_block16_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block16_1_bn[0][0]']
vation)
conv4_block16_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block16_1_relu[0][0]']
2D)
conv4_block16_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block16_2_conv[0][0]']
ormalization)
conv4_block16_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block16_2_bn[0][0]']
vation)
conv4_block16_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block16_2_relu[0][0]']
2D)
conv4_block16_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block16_3_conv[0][0]']
ormalization)
conv4_block16_add (Add) (None, 3, 3, 1024) 0 ['conv4_block15_out[0][0]',
'conv4_block16_3_bn[0][0]']
conv4_block16_out (Activat (None, 3, 3, 1024) 0 ['conv4_block16_add[0][0]']
ion)
conv4_block17_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block16_out[0][0]']
2D)
conv4_block17_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block17_1_conv[0][0]']
ormalization)
conv4_block17_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block17_1_bn[0][0]']
vation)
conv4_block17_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block17_1_relu[0][0]']
2D)
conv4_block17_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block17_2_conv[0][0]']
ormalization)
conv4_block17_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block17_2_bn[0][0]']
vation)
conv4_block17_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block17_2_relu[0][0]']
2D)
conv4_block17_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block17_3_conv[0][0]']
ormalization)
conv4_block17_add (Add) (None, 3, 3, 1024) 0 ['conv4_block16_out[0][0]',
'conv4_block17_3_bn[0][0]']
conv4_block17_out (Activat (None, 3, 3, 1024) 0 ['conv4_block17_add[0][0]']
ion)
conv4_block18_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block17_out[0][0]']
2D)
conv4_block18_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block18_1_conv[0][0]']
ormalization)
conv4_block18_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block18_1_bn[0][0]']
vation)
conv4_block18_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block18_1_relu[0][0]']
2D)
conv4_block18_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block18_2_conv[0][0]']
ormalization)
conv4_block18_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block18_2_bn[0][0]']
vation)
conv4_block18_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block18_2_relu[0][0]']
2D)
conv4_block18_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block18_3_conv[0][0]']
ormalization)
conv4_block18_add (Add) (None, 3, 3, 1024) 0 ['conv4_block17_out[0][0]',
'conv4_block18_3_bn[0][0]']
conv4_block18_out (Activat (None, 3, 3, 1024) 0 ['conv4_block18_add[0][0]']
ion)
conv4_block19_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block18_out[0][0]']
2D)
conv4_block19_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block19_1_conv[0][0]']
ormalization)
conv4_block19_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block19_1_bn[0][0]']
vation)
conv4_block19_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block19_1_relu[0][0]']
2D)
conv4_block19_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block19_2_conv[0][0]']
ormalization)
conv4_block19_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block19_2_bn[0][0]']
vation)
conv4_block19_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block19_2_relu[0][0]']
2D)
conv4_block19_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block19_3_conv[0][0]']
ormalization)
conv4_block19_add (Add) (None, 3, 3, 1024) 0 ['conv4_block18_out[0][0]',
'conv4_block19_3_bn[0][0]']
conv4_block19_out (Activat (None, 3, 3, 1024) 0 ['conv4_block19_add[0][0]']
ion)
conv4_block20_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block19_out[0][0]']
2D)
conv4_block20_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block20_1_conv[0][0]']
ormalization)
conv4_block20_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block20_1_bn[0][0]']
vation)
conv4_block20_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block20_1_relu[0][0]']
2D)
conv4_block20_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block20_2_conv[0][0]']
ormalization)
conv4_block20_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block20_2_bn[0][0]']
vation)
conv4_block20_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block20_2_relu[0][0]']
2D)
conv4_block20_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block20_3_conv[0][0]']
ormalization)
conv4_block20_add (Add) (None, 3, 3, 1024) 0 ['conv4_block19_out[0][0]',
'conv4_block20_3_bn[0][0]']
conv4_block20_out (Activat (None, 3, 3, 1024) 0 ['conv4_block20_add[0][0]']
ion)
conv4_block21_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block20_out[0][0]']
2D)
conv4_block21_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block21_1_conv[0][0]']
ormalization)
conv4_block21_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block21_1_bn[0][0]']
vation)
conv4_block21_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block21_1_relu[0][0]']
2D)
conv4_block21_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block21_2_conv[0][0]']
ormalization)
conv4_block21_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block21_2_bn[0][0]']
vation)
conv4_block21_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block21_2_relu[0][0]']
2D)
conv4_block21_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block21_3_conv[0][0]']
ormalization)
conv4_block21_add (Add) (None, 3, 3, 1024) 0 ['conv4_block20_out[0][0]',
'conv4_block21_3_bn[0][0]']
conv4_block21_out (Activat (None, 3, 3, 1024) 0 ['conv4_block21_add[0][0]']
ion)
conv4_block22_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block21_out[0][0]']
2D)
conv4_block22_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block22_1_conv[0][0]']
ormalization)
conv4_block22_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block22_1_bn[0][0]']
vation)
conv4_block22_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block22_1_relu[0][0]']
2D)
conv4_block22_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block22_2_conv[0][0]']
ormalization)
conv4_block22_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block22_2_bn[0][0]']
vation)
conv4_block22_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block22_2_relu[0][0]']
2D)
conv4_block22_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block22_3_conv[0][0]']
ormalization)
conv4_block22_add (Add) (None, 3, 3, 1024) 0 ['conv4_block21_out[0][0]',
'conv4_block22_3_bn[0][0]']
conv4_block22_out (Activat (None, 3, 3, 1024) 0 ['conv4_block22_add[0][0]']
ion)
conv4_block23_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block22_out[0][0]']
2D)
conv4_block23_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block23_1_conv[0][0]']
ormalization)
conv4_block23_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block23_1_bn[0][0]']
vation)
conv4_block23_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block23_1_relu[0][0]']
2D)
conv4_block23_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block23_2_conv[0][0]']
ormalization)
conv4_block23_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block23_2_bn[0][0]']
vation)
conv4_block23_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block23_2_relu[0][0]']
2D)
conv4_block23_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block23_3_conv[0][0]']
ormalization)
conv4_block23_add (Add) (None, 3, 3, 1024) 0 ['conv4_block22_out[0][0]',
'conv4_block23_3_bn[0][0]']
conv4_block23_out (Activat (None, 3, 3, 1024) 0 ['conv4_block23_add[0][0]']
ion)
conv5_block1_1_conv (Conv2 (None, 2, 2, 512) 524800 ['conv4_block23_out[0][0]']
D)
conv5_block1_1_bn (BatchNo (None, 2, 2, 512) 2048 ['conv5_block1_1_conv[0][0]']
rmalization)
conv5_block1_1_relu (Activ (None, 2, 2, 512) 0 ['conv5_block1_1_bn[0][0]']
ation)
conv5_block1_2_conv (Conv2 (None, 2, 2, 512) 2359808 ['conv5_block1_1_relu[0][0]']
D)
conv5_block1_2_bn (BatchNo (None, 2, 2, 512) 2048 ['conv5_block1_2_conv[0][0]']
rmalization)
conv5_block1_2_relu (Activ (None, 2, 2, 512) 0 ['conv5_block1_2_bn[0][0]']
ation)
conv5_block1_0_conv (Conv2 (None, 2, 2, 2048) 2099200 ['conv4_block23_out[0][0]']
D)
conv5_block1_3_conv (Conv2 (None, 2, 2, 2048) 1050624 ['conv5_block1_2_relu[0][0]']
D)
conv5_block1_0_bn (BatchNo (None, 2, 2, 2048) 8192 ['conv5_block1_0_conv[0][0]']
rmalization)
conv5_block1_3_bn (BatchNo (None, 2, 2, 2048) 8192 ['conv5_block1_3_conv[0][0]']
rmalization)
conv5_block1_add (Add) (None, 2, 2, 2048) 0 ['conv5_block1_0_bn[0][0]',
'conv5_block1_3_bn[0][0]']
conv5_block1_out (Activati (None, 2, 2, 2048) 0 ['conv5_block1_add[0][0]']
on)
conv5_block2_1_conv (Conv2 (None, 2, 2, 512) 1049088 ['conv5_block1_out[0][0]']
D)
conv5_block2_1_bn (BatchNo (None, 2, 2, 512) 2048 ['conv5_block2_1_conv[0][0]']
rmalization)
conv5_block2_1_relu (Activ (None, 2, 2, 512) 0 ['conv5_block2_1_bn[0][0]']
ation)
conv5_block2_2_conv (Conv2 (None, 2, 2, 512) 2359808 ['conv5_block2_1_relu[0][0]']
D)
conv5_block2_2_bn (BatchNo (None, 2, 2, 512) 2048 ['conv5_block2_2_conv[0][0]']
rmalization)
conv5_block2_2_relu (Activ (None, 2, 2, 512) 0 ['conv5_block2_2_bn[0][0]']
ation)
conv5_block2_3_conv (Conv2 (None, 2, 2, 2048) 1050624 ['conv5_block2_2_relu[0][0]']
D)
conv5_block2_3_bn (BatchNo (None, 2, 2, 2048) 8192 ['conv5_block2_3_conv[0][0]']
rmalization)
conv5_block2_add (Add) (None, 2, 2, 2048) 0 ['conv5_block1_out[0][0]',
'conv5_block2_3_bn[0][0]']
conv5_block2_out (Activati (None, 2, 2, 2048) 0 ['conv5_block2_add[0][0]']
on)
conv5_block3_1_conv (Conv2 (None, 2, 2, 512) 1049088 ['conv5_block2_out[0][0]']
D)
conv5_block3_1_bn (BatchNo (None, 2, 2, 512) 2048 ['conv5_block3_1_conv[0][0]']
rmalization)
conv5_block3_1_relu (Activ (None, 2, 2, 512) 0 ['conv5_block3_1_bn[0][0]']
ation)
conv5_block3_2_conv (Conv2 (None, 2, 2, 512) 2359808 ['conv5_block3_1_relu[0][0]']
D)
conv5_block3_2_bn (BatchNo (None, 2, 2, 512) 2048 ['conv5_block3_2_conv[0][0]']
rmalization)
conv5_block3_2_relu (Activ (None, 2, 2, 512) 0 ['conv5_block3_2_bn[0][0]']
ation)
conv5_block3_3_conv (Conv2 (None, 2, 2, 2048) 1050624 ['conv5_block3_2_relu[0][0]']
D)
conv5_block3_3_bn (BatchNo (None, 2, 2, 2048) 8192 ['conv5_block3_3_conv[0][0]']
rmalization)
conv5_block3_add (Add) (None, 2, 2, 2048) 0 ['conv5_block2_out[0][0]',
'conv5_block3_3_bn[0][0]']
conv5_block3_out (Activati (None, 2, 2, 2048) 0 ['conv5_block3_add[0][0]']
on)
==================================================================================================
Total params: 42658176 (162.73 MB)
Trainable params: 42552832 (162.33 MB)
Non-trainable params: 105344 (411.50 KB)
__________________________________________________________________________________________________
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=False,
fill_mode='nearest'
)
# Define the image size and paths
image_size = 48
train_path = "Facial_emotion_images/train"
test_path = "Facial_emotion_images/test"
validation_path = "Facial_emotion_images/validation"
# Initialize the data generator with rescaling
datagen = ImageDataGenerator(rescale=1./255)
# Prepare the data generators
train_generator = datagen.flow_from_directory(
train_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical')
validation_generator = datagen.flow_from_directory(
validation_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical')
test_generator = datagen.flow_from_directory(
test_path,
target_size=(48, 48),
shuffle=False)
Found 15109 images belonging to 4 classes. Found 4977 images belonging to 4 classes. Found 128 images belonging to 4 classes.
np.random.seed(42)
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, GlobalAveragePooling2D
from tensorflow.keras.layers import Flatten, Dense, Dropout, BatchNormalization
from tensorflow.keras.optimizers import AdamW, SGD, Adamax
# Load ResNet50 model pre-trained on ImageNet without the top layer
base_model = ResNet50(weights='imagenet', include_top=False, input_shape=(48, 48, 3))
# Add custom layers on top of the base model
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
x = Dropout(0.4)(x)
x = Dense(1024, activation='relu')(x)
x = Dropout(0.4)(x)
predictions = Dense(4, activation='softmax')(x)
resnet_model2 = Model(inputs=base_model.input, outputs=predictions)
# Compile the model
resnet_model2.compile(optimizer=Adamax(learning_rate=0.00025), loss='categorical_crossentropy', metrics=['accuracy'])
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, min_lr=0.000001)
model_checkpoint = ModelCheckpoint('best_model.h5', monitor='val_accuracy', save_best_only=True)
early_stopping = EarlyStopping(monitor='val_loss', patience=7, restore_best_weights=True)
history = resnet_model2.fit(
train_generator,
steps_per_epoch=train_generator.samples // train_generator.batch_size,
epochs=20,
validation_data=validation_generator,
validation_steps=validation_generator.samples // validation_generator.batch_size,
class_weight=class_weights_adjusted,
callbacks=[reduce_lr, model_checkpoint, early_stopping]
)
Epoch 1/20 472/472 [==============================] - ETA: 0s - loss: 1.2432 - accuracy: 0.4541
/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py:3103: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
saving_api.save_model(
472/472 [==============================] - 53s 62ms/step - loss: 1.2432 - accuracy: 0.4541 - val_loss: 148.1311 - val_accuracy: 0.2230 - lr: 2.5000e-04 Epoch 2/20 472/472 [==============================] - 27s 57ms/step - loss: 0.8767 - accuracy: 0.6330 - val_loss: 1.2080 - val_accuracy: 0.4980 - lr: 2.5000e-04 Epoch 3/20 472/472 [==============================] - 27s 57ms/step - loss: 0.6773 - accuracy: 0.7215 - val_loss: 0.8134 - val_accuracy: 0.6853 - lr: 2.5000e-04 Epoch 4/20 472/472 [==============================] - 26s 55ms/step - loss: 0.5270 - accuracy: 0.7853 - val_loss: 0.8374 - val_accuracy: 0.6808 - lr: 2.5000e-04 Epoch 5/20 472/472 [==============================] - 27s 58ms/step - loss: 0.3790 - accuracy: 0.8498 - val_loss: 0.8913 - val_accuracy: 0.7121 - lr: 2.5000e-04 Epoch 6/20 472/472 [==============================] - 26s 55ms/step - loss: 0.2578 - accuracy: 0.9021 - val_loss: 1.2126 - val_accuracy: 0.6875 - lr: 2.5000e-04 Epoch 7/20 472/472 [==============================] - 27s 57ms/step - loss: 0.1923 - accuracy: 0.9292 - val_loss: 0.9888 - val_accuracy: 0.7210 - lr: 2.5000e-04 Epoch 8/20 472/472 [==============================] - 26s 54ms/step - loss: 0.1055 - accuracy: 0.9625 - val_loss: 1.2811 - val_accuracy: 0.7185 - lr: 2.5000e-04 Epoch 9/20 472/472 [==============================] - 27s 57ms/step - loss: 0.0546 - accuracy: 0.9821 - val_loss: 1.2760 - val_accuracy: 0.7306 - lr: 5.0000e-05 Epoch 10/20 472/472 [==============================] - 27s 57ms/step - loss: 0.0351 - accuracy: 0.9884 - val_loss: 1.3550 - val_accuracy: 0.7310 - lr: 5.0000e-05
from sklearn.metrics import confusion_matrix, classification_report
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(test_generator.classes, np.argmax(resnet_model2.predict(test_generator), axis=1))
# Confusion matrix
print("Confusion Matrix:")
print(cm)
# Classification report
# Plot the training and validation accuracy values
plt.figure(figsize=(15, 5))
plt.subplot(1, 3, 1)
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('Model_resnet Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Plot the training and validation loss values
plt.subplot(1, 3, 2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Model_resnet Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Plot the training and validation loss values
plt.subplot(1, 3, 3)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", cbar=False)
plt.xlabel("Predicted Label")
plt.ylabel("True Label")
plt.title("Confusion Matrix for Resnet Model")
plt.show()
predicted_labels = np.argmax(resnet_model2.predict(test_generator), axis=1)
# Get the true labels for the test set
true_labels = test_generator.classes
# Get the class names
class_names_list = test_generator.class_indices.keys()
print(class_names_list)
# Calculate and display the metrics
metrics_score(true_labels, predicted_labels)
# Plot the confusion matrix
cm = confusion_matrix(true_labels, predicted_labels)
4/4 [==============================] - 1s 20ms/step Confusion Matrix: [[26 2 2 2] [ 2 25 5 0] [ 1 12 17 2] [ 0 3 1 28]]
4/4 [==============================] - 0s 22ms/step
dict_keys(['happy', 'neutral', 'sad', 'surprise'])
precision recall f1-score support
0 0.90 0.81 0.85 32
1 0.60 0.78 0.68 32
2 0.68 0.53 0.60 32
3 0.88 0.88 0.88 32
accuracy 0.75 128
macro avg 0.76 0.75 0.75 128
weighted avg 0.76 0.75 0.75 128
# Evaluate the model on the test set
val_loss, val_accuracy = resnet_model2.evaluate(
test_generator,
steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
The ResNet50-based custom model, adapted for classifying four emotional states, achieved an overall accuracy of 75%. The model's precision and recall suggest it performs best in recognizing 'happy' and 'surprise' emotions, with scores indicating high reliability in these predictions. However, it shows a need for improvement in accurately classifying 'neutral' and 'sad' states, as evidenced by lower precision for 'neutral' and lower recall for 'sad'. The confusion matrix details the model's performance across all classes, highlighting specific areas where misclassifications occur, particularly in distinguishing 'neutral' from 'sad'. Optimizations could focus on enhancing feature extraction and addressing class imbalance to improve accuracy and balance across all emotion categories.
np.random.seed(42)
import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.applications.efficientnet import EfficientNetB0, preprocess_input
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D
from tensorflow.keras import regularizers
import tensorflow as tf
import tensorflow.keras.applications as ap
EfficientNet = ap.EfficientNetV2B2(include_top=False,weights="imagenet", input_shape= (48, 48, 3))
EfficientNet.summary()
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/efficientnet_v2/efficientnetv2-b2_notop.h5
35839040/35839040 [==============================] - 3s 0us/step
Model: "efficientnetv2-b2"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) [(None, 48, 48, 3)] 0 []
rescaling (Rescaling) (None, 48, 48, 3) 0 ['input_1[0][0]']
normalization (Normalizati (None, 48, 48, 3) 0 ['rescaling[0][0]']
on)
stem_conv (Conv2D) (None, 24, 24, 32) 864 ['normalization[0][0]']
stem_bn (BatchNormalizatio (None, 24, 24, 32) 128 ['stem_conv[0][0]']
n)
stem_activation (Activatio (None, 24, 24, 32) 0 ['stem_bn[0][0]']
n)
block1a_project_conv (Conv (None, 24, 24, 16) 4608 ['stem_activation[0][0]']
2D)
block1a_project_bn (BatchN (None, 24, 24, 16) 64 ['block1a_project_conv[0][0]']
ormalization)
block1a_project_activation (None, 24, 24, 16) 0 ['block1a_project_bn[0][0]']
(Activation)
block1b_project_conv (Conv (None, 24, 24, 16) 2304 ['block1a_project_activation[0
2D) ][0]']
block1b_project_bn (BatchN (None, 24, 24, 16) 64 ['block1b_project_conv[0][0]']
ormalization)
block1b_project_activation (None, 24, 24, 16) 0 ['block1b_project_bn[0][0]']
(Activation)
block1b_drop (Dropout) (None, 24, 24, 16) 0 ['block1b_project_activation[0
][0]']
block1b_add (Add) (None, 24, 24, 16) 0 ['block1b_drop[0][0]',
'block1a_project_activation[0
][0]']
block2a_expand_conv (Conv2 (None, 12, 12, 64) 9216 ['block1b_add[0][0]']
D)
block2a_expand_bn (BatchNo (None, 12, 12, 64) 256 ['block2a_expand_conv[0][0]']
rmalization)
block2a_expand_activation (None, 12, 12, 64) 0 ['block2a_expand_bn[0][0]']
(Activation)
block2a_project_conv (Conv (None, 12, 12, 32) 2048 ['block2a_expand_activation[0]
2D) [0]']
block2a_project_bn (BatchN (None, 12, 12, 32) 128 ['block2a_project_conv[0][0]']
ormalization)
block2b_expand_conv (Conv2 (None, 12, 12, 128) 36864 ['block2a_project_bn[0][0]']
D)
block2b_expand_bn (BatchNo (None, 12, 12, 128) 512 ['block2b_expand_conv[0][0]']
rmalization)
block2b_expand_activation (None, 12, 12, 128) 0 ['block2b_expand_bn[0][0]']
(Activation)
block2b_project_conv (Conv (None, 12, 12, 32) 4096 ['block2b_expand_activation[0]
2D) [0]']
block2b_project_bn (BatchN (None, 12, 12, 32) 128 ['block2b_project_conv[0][0]']
ormalization)
block2b_drop (Dropout) (None, 12, 12, 32) 0 ['block2b_project_bn[0][0]']
block2b_add (Add) (None, 12, 12, 32) 0 ['block2b_drop[0][0]',
'block2a_project_bn[0][0]']
block2c_expand_conv (Conv2 (None, 12, 12, 128) 36864 ['block2b_add[0][0]']
D)
block2c_expand_bn (BatchNo (None, 12, 12, 128) 512 ['block2c_expand_conv[0][0]']
rmalization)
block2c_expand_activation (None, 12, 12, 128) 0 ['block2c_expand_bn[0][0]']
(Activation)
block2c_project_conv (Conv (None, 12, 12, 32) 4096 ['block2c_expand_activation[0]
2D) [0]']
block2c_project_bn (BatchN (None, 12, 12, 32) 128 ['block2c_project_conv[0][0]']
ormalization)
block2c_drop (Dropout) (None, 12, 12, 32) 0 ['block2c_project_bn[0][0]']
block2c_add (Add) (None, 12, 12, 32) 0 ['block2c_drop[0][0]',
'block2b_add[0][0]']
block3a_expand_conv (Conv2 (None, 6, 6, 128) 36864 ['block2c_add[0][0]']
D)
block3a_expand_bn (BatchNo (None, 6, 6, 128) 512 ['block3a_expand_conv[0][0]']
rmalization)
block3a_expand_activation (None, 6, 6, 128) 0 ['block3a_expand_bn[0][0]']
(Activation)
block3a_project_conv (Conv (None, 6, 6, 56) 7168 ['block3a_expand_activation[0]
2D) [0]']
block3a_project_bn (BatchN (None, 6, 6, 56) 224 ['block3a_project_conv[0][0]']
ormalization)
block3b_expand_conv (Conv2 (None, 6, 6, 224) 112896 ['block3a_project_bn[0][0]']
D)
block3b_expand_bn (BatchNo (None, 6, 6, 224) 896 ['block3b_expand_conv[0][0]']
rmalization)
block3b_expand_activation (None, 6, 6, 224) 0 ['block3b_expand_bn[0][0]']
(Activation)
block3b_project_conv (Conv (None, 6, 6, 56) 12544 ['block3b_expand_activation[0]
2D) [0]']
block3b_project_bn (BatchN (None, 6, 6, 56) 224 ['block3b_project_conv[0][0]']
ormalization)
block3b_drop (Dropout) (None, 6, 6, 56) 0 ['block3b_project_bn[0][0]']
block3b_add (Add) (None, 6, 6, 56) 0 ['block3b_drop[0][0]',
'block3a_project_bn[0][0]']
block3c_expand_conv (Conv2 (None, 6, 6, 224) 112896 ['block3b_add[0][0]']
D)
block3c_expand_bn (BatchNo (None, 6, 6, 224) 896 ['block3c_expand_conv[0][0]']
rmalization)
block3c_expand_activation (None, 6, 6, 224) 0 ['block3c_expand_bn[0][0]']
(Activation)
block3c_project_conv (Conv (None, 6, 6, 56) 12544 ['block3c_expand_activation[0]
2D) [0]']
block3c_project_bn (BatchN (None, 6, 6, 56) 224 ['block3c_project_conv[0][0]']
ormalization)
block3c_drop (Dropout) (None, 6, 6, 56) 0 ['block3c_project_bn[0][0]']
block3c_add (Add) (None, 6, 6, 56) 0 ['block3c_drop[0][0]',
'block3b_add[0][0]']
block4a_expand_conv (Conv2 (None, 6, 6, 224) 12544 ['block3c_add[0][0]']
D)
block4a_expand_bn (BatchNo (None, 6, 6, 224) 896 ['block4a_expand_conv[0][0]']
rmalization)
block4a_expand_activation (None, 6, 6, 224) 0 ['block4a_expand_bn[0][0]']
(Activation)
block4a_dwconv2 (Depthwise (None, 3, 3, 224) 2016 ['block4a_expand_activation[0]
Conv2D) [0]']
block4a_bn (BatchNormaliza (None, 3, 3, 224) 896 ['block4a_dwconv2[0][0]']
tion)
block4a_activation (Activa (None, 3, 3, 224) 0 ['block4a_bn[0][0]']
tion)
block4a_se_squeeze (Global (None, 224) 0 ['block4a_activation[0][0]']
AveragePooling2D)
block4a_se_reshape (Reshap (None, 1, 1, 224) 0 ['block4a_se_squeeze[0][0]']
e)
block4a_se_reduce (Conv2D) (None, 1, 1, 14) 3150 ['block4a_se_reshape[0][0]']
block4a_se_expand (Conv2D) (None, 1, 1, 224) 3360 ['block4a_se_reduce[0][0]']
block4a_se_excite (Multipl (None, 3, 3, 224) 0 ['block4a_activation[0][0]',
y) 'block4a_se_expand[0][0]']
block4a_project_conv (Conv (None, 3, 3, 104) 23296 ['block4a_se_excite[0][0]']
2D)
block4a_project_bn (BatchN (None, 3, 3, 104) 416 ['block4a_project_conv[0][0]']
ormalization)
block4b_expand_conv (Conv2 (None, 3, 3, 416) 43264 ['block4a_project_bn[0][0]']
D)
block4b_expand_bn (BatchNo (None, 3, 3, 416) 1664 ['block4b_expand_conv[0][0]']
rmalization)
block4b_expand_activation (None, 3, 3, 416) 0 ['block4b_expand_bn[0][0]']
(Activation)
block4b_dwconv2 (Depthwise (None, 3, 3, 416) 3744 ['block4b_expand_activation[0]
Conv2D) [0]']
block4b_bn (BatchNormaliza (None, 3, 3, 416) 1664 ['block4b_dwconv2[0][0]']
tion)
block4b_activation (Activa (None, 3, 3, 416) 0 ['block4b_bn[0][0]']
tion)
block4b_se_squeeze (Global (None, 416) 0 ['block4b_activation[0][0]']
AveragePooling2D)
block4b_se_reshape (Reshap (None, 1, 1, 416) 0 ['block4b_se_squeeze[0][0]']
e)
block4b_se_reduce (Conv2D) (None, 1, 1, 26) 10842 ['block4b_se_reshape[0][0]']
block4b_se_expand (Conv2D) (None, 1, 1, 416) 11232 ['block4b_se_reduce[0][0]']
block4b_se_excite (Multipl (None, 3, 3, 416) 0 ['block4b_activation[0][0]',
y) 'block4b_se_expand[0][0]']
block4b_project_conv (Conv (None, 3, 3, 104) 43264 ['block4b_se_excite[0][0]']
2D)
block4b_project_bn (BatchN (None, 3, 3, 104) 416 ['block4b_project_conv[0][0]']
ormalization)
block4b_drop (Dropout) (None, 3, 3, 104) 0 ['block4b_project_bn[0][0]']
block4b_add (Add) (None, 3, 3, 104) 0 ['block4b_drop[0][0]',
'block4a_project_bn[0][0]']
block4c_expand_conv (Conv2 (None, 3, 3, 416) 43264 ['block4b_add[0][0]']
D)
block4c_expand_bn (BatchNo (None, 3, 3, 416) 1664 ['block4c_expand_conv[0][0]']
rmalization)
block4c_expand_activation (None, 3, 3, 416) 0 ['block4c_expand_bn[0][0]']
(Activation)
block4c_dwconv2 (Depthwise (None, 3, 3, 416) 3744 ['block4c_expand_activation[0]
Conv2D) [0]']
block4c_bn (BatchNormaliza (None, 3, 3, 416) 1664 ['block4c_dwconv2[0][0]']
tion)
block4c_activation (Activa (None, 3, 3, 416) 0 ['block4c_bn[0][0]']
tion)
block4c_se_squeeze (Global (None, 416) 0 ['block4c_activation[0][0]']
AveragePooling2D)
block4c_se_reshape (Reshap (None, 1, 1, 416) 0 ['block4c_se_squeeze[0][0]']
e)
block4c_se_reduce (Conv2D) (None, 1, 1, 26) 10842 ['block4c_se_reshape[0][0]']
block4c_se_expand (Conv2D) (None, 1, 1, 416) 11232 ['block4c_se_reduce[0][0]']
block4c_se_excite (Multipl (None, 3, 3, 416) 0 ['block4c_activation[0][0]',
y) 'block4c_se_expand[0][0]']
block4c_project_conv (Conv (None, 3, 3, 104) 43264 ['block4c_se_excite[0][0]']
2D)
block4c_project_bn (BatchN (None, 3, 3, 104) 416 ['block4c_project_conv[0][0]']
ormalization)
block4c_drop (Dropout) (None, 3, 3, 104) 0 ['block4c_project_bn[0][0]']
block4c_add (Add) (None, 3, 3, 104) 0 ['block4c_drop[0][0]',
'block4b_add[0][0]']
block4d_expand_conv (Conv2 (None, 3, 3, 416) 43264 ['block4c_add[0][0]']
D)
block4d_expand_bn (BatchNo (None, 3, 3, 416) 1664 ['block4d_expand_conv[0][0]']
rmalization)
block4d_expand_activation (None, 3, 3, 416) 0 ['block4d_expand_bn[0][0]']
(Activation)
block4d_dwconv2 (Depthwise (None, 3, 3, 416) 3744 ['block4d_expand_activation[0]
Conv2D) [0]']
block4d_bn (BatchNormaliza (None, 3, 3, 416) 1664 ['block4d_dwconv2[0][0]']
tion)
block4d_activation (Activa (None, 3, 3, 416) 0 ['block4d_bn[0][0]']
tion)
block4d_se_squeeze (Global (None, 416) 0 ['block4d_activation[0][0]']
AveragePooling2D)
block4d_se_reshape (Reshap (None, 1, 1, 416) 0 ['block4d_se_squeeze[0][0]']
e)
block4d_se_reduce (Conv2D) (None, 1, 1, 26) 10842 ['block4d_se_reshape[0][0]']
block4d_se_expand (Conv2D) (None, 1, 1, 416) 11232 ['block4d_se_reduce[0][0]']
block4d_se_excite (Multipl (None, 3, 3, 416) 0 ['block4d_activation[0][0]',
y) 'block4d_se_expand[0][0]']
block4d_project_conv (Conv (None, 3, 3, 104) 43264 ['block4d_se_excite[0][0]']
2D)
block4d_project_bn (BatchN (None, 3, 3, 104) 416 ['block4d_project_conv[0][0]']
ormalization)
block4d_drop (Dropout) (None, 3, 3, 104) 0 ['block4d_project_bn[0][0]']
block4d_add (Add) (None, 3, 3, 104) 0 ['block4d_drop[0][0]',
'block4c_add[0][0]']
block5a_expand_conv (Conv2 (None, 3, 3, 624) 64896 ['block4d_add[0][0]']
D)
block5a_expand_bn (BatchNo (None, 3, 3, 624) 2496 ['block5a_expand_conv[0][0]']
rmalization)
block5a_expand_activation (None, 3, 3, 624) 0 ['block5a_expand_bn[0][0]']
(Activation)
block5a_dwconv2 (Depthwise (None, 3, 3, 624) 5616 ['block5a_expand_activation[0]
Conv2D) [0]']
block5a_bn (BatchNormaliza (None, 3, 3, 624) 2496 ['block5a_dwconv2[0][0]']
tion)
block5a_activation (Activa (None, 3, 3, 624) 0 ['block5a_bn[0][0]']
tion)
block5a_se_squeeze (Global (None, 624) 0 ['block5a_activation[0][0]']
AveragePooling2D)
block5a_se_reshape (Reshap (None, 1, 1, 624) 0 ['block5a_se_squeeze[0][0]']
e)
block5a_se_reduce (Conv2D) (None, 1, 1, 26) 16250 ['block5a_se_reshape[0][0]']
block5a_se_expand (Conv2D) (None, 1, 1, 624) 16848 ['block5a_se_reduce[0][0]']
block5a_se_excite (Multipl (None, 3, 3, 624) 0 ['block5a_activation[0][0]',
y) 'block5a_se_expand[0][0]']
block5a_project_conv (Conv (None, 3, 3, 120) 74880 ['block5a_se_excite[0][0]']
2D)
block5a_project_bn (BatchN (None, 3, 3, 120) 480 ['block5a_project_conv[0][0]']
ormalization)
block5b_expand_conv (Conv2 (None, 3, 3, 720) 86400 ['block5a_project_bn[0][0]']
D)
block5b_expand_bn (BatchNo (None, 3, 3, 720) 2880 ['block5b_expand_conv[0][0]']
rmalization)
block5b_expand_activation (None, 3, 3, 720) 0 ['block5b_expand_bn[0][0]']
(Activation)
block5b_dwconv2 (Depthwise (None, 3, 3, 720) 6480 ['block5b_expand_activation[0]
Conv2D) [0]']
block5b_bn (BatchNormaliza (None, 3, 3, 720) 2880 ['block5b_dwconv2[0][0]']
tion)
block5b_activation (Activa (None, 3, 3, 720) 0 ['block5b_bn[0][0]']
tion)
block5b_se_squeeze (Global (None, 720) 0 ['block5b_activation[0][0]']
AveragePooling2D)
block5b_se_reshape (Reshap (None, 1, 1, 720) 0 ['block5b_se_squeeze[0][0]']
e)
block5b_se_reduce (Conv2D) (None, 1, 1, 30) 21630 ['block5b_se_reshape[0][0]']
block5b_se_expand (Conv2D) (None, 1, 1, 720) 22320 ['block5b_se_reduce[0][0]']
block5b_se_excite (Multipl (None, 3, 3, 720) 0 ['block5b_activation[0][0]',
y) 'block5b_se_expand[0][0]']
block5b_project_conv (Conv (None, 3, 3, 120) 86400 ['block5b_se_excite[0][0]']
2D)
block5b_project_bn (BatchN (None, 3, 3, 120) 480 ['block5b_project_conv[0][0]']
ormalization)
block5b_drop (Dropout) (None, 3, 3, 120) 0 ['block5b_project_bn[0][0]']
block5b_add (Add) (None, 3, 3, 120) 0 ['block5b_drop[0][0]',
'block5a_project_bn[0][0]']
block5c_expand_conv (Conv2 (None, 3, 3, 720) 86400 ['block5b_add[0][0]']
D)
block5c_expand_bn (BatchNo (None, 3, 3, 720) 2880 ['block5c_expand_conv[0][0]']
rmalization)
block5c_expand_activation (None, 3, 3, 720) 0 ['block5c_expand_bn[0][0]']
(Activation)
block5c_dwconv2 (Depthwise (None, 3, 3, 720) 6480 ['block5c_expand_activation[0]
Conv2D) [0]']
block5c_bn (BatchNormaliza (None, 3, 3, 720) 2880 ['block5c_dwconv2[0][0]']
tion)
block5c_activation (Activa (None, 3, 3, 720) 0 ['block5c_bn[0][0]']
tion)
block5c_se_squeeze (Global (None, 720) 0 ['block5c_activation[0][0]']
AveragePooling2D)
block5c_se_reshape (Reshap (None, 1, 1, 720) 0 ['block5c_se_squeeze[0][0]']
e)
block5c_se_reduce (Conv2D) (None, 1, 1, 30) 21630 ['block5c_se_reshape[0][0]']
block5c_se_expand (Conv2D) (None, 1, 1, 720) 22320 ['block5c_se_reduce[0][0]']
block5c_se_excite (Multipl (None, 3, 3, 720) 0 ['block5c_activation[0][0]',
y) 'block5c_se_expand[0][0]']
block5c_project_conv (Conv (None, 3, 3, 120) 86400 ['block5c_se_excite[0][0]']
2D)
block5c_project_bn (BatchN (None, 3, 3, 120) 480 ['block5c_project_conv[0][0]']
ormalization)
block5c_drop (Dropout) (None, 3, 3, 120) 0 ['block5c_project_bn[0][0]']
block5c_add (Add) (None, 3, 3, 120) 0 ['block5c_drop[0][0]',
'block5b_add[0][0]']
block5d_expand_conv (Conv2 (None, 3, 3, 720) 86400 ['block5c_add[0][0]']
D)
block5d_expand_bn (BatchNo (None, 3, 3, 720) 2880 ['block5d_expand_conv[0][0]']
rmalization)
block5d_expand_activation (None, 3, 3, 720) 0 ['block5d_expand_bn[0][0]']
(Activation)
block5d_dwconv2 (Depthwise (None, 3, 3, 720) 6480 ['block5d_expand_activation[0]
Conv2D) [0]']
block5d_bn (BatchNormaliza (None, 3, 3, 720) 2880 ['block5d_dwconv2[0][0]']
tion)
block5d_activation (Activa (None, 3, 3, 720) 0 ['block5d_bn[0][0]']
tion)
block5d_se_squeeze (Global (None, 720) 0 ['block5d_activation[0][0]']
AveragePooling2D)
block5d_se_reshape (Reshap (None, 1, 1, 720) 0 ['block5d_se_squeeze[0][0]']
e)
block5d_se_reduce (Conv2D) (None, 1, 1, 30) 21630 ['block5d_se_reshape[0][0]']
block5d_se_expand (Conv2D) (None, 1, 1, 720) 22320 ['block5d_se_reduce[0][0]']
block5d_se_excite (Multipl (None, 3, 3, 720) 0 ['block5d_activation[0][0]',
y) 'block5d_se_expand[0][0]']
block5d_project_conv (Conv (None, 3, 3, 120) 86400 ['block5d_se_excite[0][0]']
2D)
block5d_project_bn (BatchN (None, 3, 3, 120) 480 ['block5d_project_conv[0][0]']
ormalization)
block5d_drop (Dropout) (None, 3, 3, 120) 0 ['block5d_project_bn[0][0]']
block5d_add (Add) (None, 3, 3, 120) 0 ['block5d_drop[0][0]',
'block5c_add[0][0]']
block5e_expand_conv (Conv2 (None, 3, 3, 720) 86400 ['block5d_add[0][0]']
D)
block5e_expand_bn (BatchNo (None, 3, 3, 720) 2880 ['block5e_expand_conv[0][0]']
rmalization)
block5e_expand_activation (None, 3, 3, 720) 0 ['block5e_expand_bn[0][0]']
(Activation)
block5e_dwconv2 (Depthwise (None, 3, 3, 720) 6480 ['block5e_expand_activation[0]
Conv2D) [0]']
block5e_bn (BatchNormaliza (None, 3, 3, 720) 2880 ['block5e_dwconv2[0][0]']
tion)
block5e_activation (Activa (None, 3, 3, 720) 0 ['block5e_bn[0][0]']
tion)
block5e_se_squeeze (Global (None, 720) 0 ['block5e_activation[0][0]']
AveragePooling2D)
block5e_se_reshape (Reshap (None, 1, 1, 720) 0 ['block5e_se_squeeze[0][0]']
e)
block5e_se_reduce (Conv2D) (None, 1, 1, 30) 21630 ['block5e_se_reshape[0][0]']
block5e_se_expand (Conv2D) (None, 1, 1, 720) 22320 ['block5e_se_reduce[0][0]']
block5e_se_excite (Multipl (None, 3, 3, 720) 0 ['block5e_activation[0][0]',
y) 'block5e_se_expand[0][0]']
block5e_project_conv (Conv (None, 3, 3, 120) 86400 ['block5e_se_excite[0][0]']
2D)
block5e_project_bn (BatchN (None, 3, 3, 120) 480 ['block5e_project_conv[0][0]']
ormalization)
block5e_drop (Dropout) (None, 3, 3, 120) 0 ['block5e_project_bn[0][0]']
block5e_add (Add) (None, 3, 3, 120) 0 ['block5e_drop[0][0]',
'block5d_add[0][0]']
block5f_expand_conv (Conv2 (None, 3, 3, 720) 86400 ['block5e_add[0][0]']
D)
block5f_expand_bn (BatchNo (None, 3, 3, 720) 2880 ['block5f_expand_conv[0][0]']
rmalization)
block5f_expand_activation (None, 3, 3, 720) 0 ['block5f_expand_bn[0][0]']
(Activation)
block5f_dwconv2 (Depthwise (None, 3, 3, 720) 6480 ['block5f_expand_activation[0]
Conv2D) [0]']
block5f_bn (BatchNormaliza (None, 3, 3, 720) 2880 ['block5f_dwconv2[0][0]']
tion)
block5f_activation (Activa (None, 3, 3, 720) 0 ['block5f_bn[0][0]']
tion)
block5f_se_squeeze (Global (None, 720) 0 ['block5f_activation[0][0]']
AveragePooling2D)
block5f_se_reshape (Reshap (None, 1, 1, 720) 0 ['block5f_se_squeeze[0][0]']
e)
block5f_se_reduce (Conv2D) (None, 1, 1, 30) 21630 ['block5f_se_reshape[0][0]']
block5f_se_expand (Conv2D) (None, 1, 1, 720) 22320 ['block5f_se_reduce[0][0]']
block5f_se_excite (Multipl (None, 3, 3, 720) 0 ['block5f_activation[0][0]',
y) 'block5f_se_expand[0][0]']
block5f_project_conv (Conv (None, 3, 3, 120) 86400 ['block5f_se_excite[0][0]']
2D)
block5f_project_bn (BatchN (None, 3, 3, 120) 480 ['block5f_project_conv[0][0]']
ormalization)
block5f_drop (Dropout) (None, 3, 3, 120) 0 ['block5f_project_bn[0][0]']
block5f_add (Add) (None, 3, 3, 120) 0 ['block5f_drop[0][0]',
'block5e_add[0][0]']
block6a_expand_conv (Conv2 (None, 3, 3, 720) 86400 ['block5f_add[0][0]']
D)
block6a_expand_bn (BatchNo (None, 3, 3, 720) 2880 ['block6a_expand_conv[0][0]']
rmalization)
block6a_expand_activation (None, 3, 3, 720) 0 ['block6a_expand_bn[0][0]']
(Activation)
block6a_dwconv2 (Depthwise (None, 2, 2, 720) 6480 ['block6a_expand_activation[0]
Conv2D) [0]']
block6a_bn (BatchNormaliza (None, 2, 2, 720) 2880 ['block6a_dwconv2[0][0]']
tion)
block6a_activation (Activa (None, 2, 2, 720) 0 ['block6a_bn[0][0]']
tion)
block6a_se_squeeze (Global (None, 720) 0 ['block6a_activation[0][0]']
AveragePooling2D)
block6a_se_reshape (Reshap (None, 1, 1, 720) 0 ['block6a_se_squeeze[0][0]']
e)
block6a_se_reduce (Conv2D) (None, 1, 1, 30) 21630 ['block6a_se_reshape[0][0]']
block6a_se_expand (Conv2D) (None, 1, 1, 720) 22320 ['block6a_se_reduce[0][0]']
block6a_se_excite (Multipl (None, 2, 2, 720) 0 ['block6a_activation[0][0]',
y) 'block6a_se_expand[0][0]']
block6a_project_conv (Conv (None, 2, 2, 208) 149760 ['block6a_se_excite[0][0]']
2D)
block6a_project_bn (BatchN (None, 2, 2, 208) 832 ['block6a_project_conv[0][0]']
ormalization)
block6b_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6a_project_bn[0][0]']
D)
block6b_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6b_expand_conv[0][0]']
rmalization)
block6b_expand_activation (None, 2, 2, 1248) 0 ['block6b_expand_bn[0][0]']
(Activation)
block6b_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6b_expand_activation[0]
Conv2D) [0]']
block6b_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6b_dwconv2[0][0]']
tion)
block6b_activation (Activa (None, 2, 2, 1248) 0 ['block6b_bn[0][0]']
tion)
block6b_se_squeeze (Global (None, 1248) 0 ['block6b_activation[0][0]']
AveragePooling2D)
block6b_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6b_se_squeeze[0][0]']
e)
block6b_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6b_se_reshape[0][0]']
block6b_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6b_se_reduce[0][0]']
block6b_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6b_activation[0][0]',
y) 'block6b_se_expand[0][0]']
block6b_project_conv (Conv (None, 2, 2, 208) 259584 ['block6b_se_excite[0][0]']
2D)
block6b_project_bn (BatchN (None, 2, 2, 208) 832 ['block6b_project_conv[0][0]']
ormalization)
block6b_drop (Dropout) (None, 2, 2, 208) 0 ['block6b_project_bn[0][0]']
block6b_add (Add) (None, 2, 2, 208) 0 ['block6b_drop[0][0]',
'block6a_project_bn[0][0]']
block6c_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6b_add[0][0]']
D)
block6c_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6c_expand_conv[0][0]']
rmalization)
block6c_expand_activation (None, 2, 2, 1248) 0 ['block6c_expand_bn[0][0]']
(Activation)
block6c_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6c_expand_activation[0]
Conv2D) [0]']
block6c_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6c_dwconv2[0][0]']
tion)
block6c_activation (Activa (None, 2, 2, 1248) 0 ['block6c_bn[0][0]']
tion)
block6c_se_squeeze (Global (None, 1248) 0 ['block6c_activation[0][0]']
AveragePooling2D)
block6c_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6c_se_squeeze[0][0]']
e)
block6c_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6c_se_reshape[0][0]']
block6c_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6c_se_reduce[0][0]']
block6c_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6c_activation[0][0]',
y) 'block6c_se_expand[0][0]']
block6c_project_conv (Conv (None, 2, 2, 208) 259584 ['block6c_se_excite[0][0]']
2D)
block6c_project_bn (BatchN (None, 2, 2, 208) 832 ['block6c_project_conv[0][0]']
ormalization)
block6c_drop (Dropout) (None, 2, 2, 208) 0 ['block6c_project_bn[0][0]']
block6c_add (Add) (None, 2, 2, 208) 0 ['block6c_drop[0][0]',
'block6b_add[0][0]']
block6d_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6c_add[0][0]']
D)
block6d_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6d_expand_conv[0][0]']
rmalization)
block6d_expand_activation (None, 2, 2, 1248) 0 ['block6d_expand_bn[0][0]']
(Activation)
block6d_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6d_expand_activation[0]
Conv2D) [0]']
block6d_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6d_dwconv2[0][0]']
tion)
block6d_activation (Activa (None, 2, 2, 1248) 0 ['block6d_bn[0][0]']
tion)
block6d_se_squeeze (Global (None, 1248) 0 ['block6d_activation[0][0]']
AveragePooling2D)
block6d_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6d_se_squeeze[0][0]']
e)
block6d_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6d_se_reshape[0][0]']
block6d_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6d_se_reduce[0][0]']
block6d_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6d_activation[0][0]',
y) 'block6d_se_expand[0][0]']
block6d_project_conv (Conv (None, 2, 2, 208) 259584 ['block6d_se_excite[0][0]']
2D)
block6d_project_bn (BatchN (None, 2, 2, 208) 832 ['block6d_project_conv[0][0]']
ormalization)
block6d_drop (Dropout) (None, 2, 2, 208) 0 ['block6d_project_bn[0][0]']
block6d_add (Add) (None, 2, 2, 208) 0 ['block6d_drop[0][0]',
'block6c_add[0][0]']
block6e_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6d_add[0][0]']
D)
block6e_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6e_expand_conv[0][0]']
rmalization)
block6e_expand_activation (None, 2, 2, 1248) 0 ['block6e_expand_bn[0][0]']
(Activation)
block6e_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6e_expand_activation[0]
Conv2D) [0]']
block6e_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6e_dwconv2[0][0]']
tion)
block6e_activation (Activa (None, 2, 2, 1248) 0 ['block6e_bn[0][0]']
tion)
block6e_se_squeeze (Global (None, 1248) 0 ['block6e_activation[0][0]']
AveragePooling2D)
block6e_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6e_se_squeeze[0][0]']
e)
block6e_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6e_se_reshape[0][0]']
block6e_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6e_se_reduce[0][0]']
block6e_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6e_activation[0][0]',
y) 'block6e_se_expand[0][0]']
block6e_project_conv (Conv (None, 2, 2, 208) 259584 ['block6e_se_excite[0][0]']
2D)
block6e_project_bn (BatchN (None, 2, 2, 208) 832 ['block6e_project_conv[0][0]']
ormalization)
block6e_drop (Dropout) (None, 2, 2, 208) 0 ['block6e_project_bn[0][0]']
block6e_add (Add) (None, 2, 2, 208) 0 ['block6e_drop[0][0]',
'block6d_add[0][0]']
block6f_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6e_add[0][0]']
D)
block6f_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6f_expand_conv[0][0]']
rmalization)
block6f_expand_activation (None, 2, 2, 1248) 0 ['block6f_expand_bn[0][0]']
(Activation)
block6f_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6f_expand_activation[0]
Conv2D) [0]']
block6f_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6f_dwconv2[0][0]']
tion)
block6f_activation (Activa (None, 2, 2, 1248) 0 ['block6f_bn[0][0]']
tion)
block6f_se_squeeze (Global (None, 1248) 0 ['block6f_activation[0][0]']
AveragePooling2D)
block6f_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6f_se_squeeze[0][0]']
e)
block6f_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6f_se_reshape[0][0]']
block6f_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6f_se_reduce[0][0]']
block6f_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6f_activation[0][0]',
y) 'block6f_se_expand[0][0]']
block6f_project_conv (Conv (None, 2, 2, 208) 259584 ['block6f_se_excite[0][0]']
2D)
block6f_project_bn (BatchN (None, 2, 2, 208) 832 ['block6f_project_conv[0][0]']
ormalization)
block6f_drop (Dropout) (None, 2, 2, 208) 0 ['block6f_project_bn[0][0]']
block6f_add (Add) (None, 2, 2, 208) 0 ['block6f_drop[0][0]',
'block6e_add[0][0]']
block6g_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6f_add[0][0]']
D)
block6g_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6g_expand_conv[0][0]']
rmalization)
block6g_expand_activation (None, 2, 2, 1248) 0 ['block6g_expand_bn[0][0]']
(Activation)
block6g_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6g_expand_activation[0]
Conv2D) [0]']
block6g_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6g_dwconv2[0][0]']
tion)
block6g_activation (Activa (None, 2, 2, 1248) 0 ['block6g_bn[0][0]']
tion)
block6g_se_squeeze (Global (None, 1248) 0 ['block6g_activation[0][0]']
AveragePooling2D)
block6g_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6g_se_squeeze[0][0]']
e)
block6g_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6g_se_reshape[0][0]']
block6g_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6g_se_reduce[0][0]']
block6g_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6g_activation[0][0]',
y) 'block6g_se_expand[0][0]']
block6g_project_conv (Conv (None, 2, 2, 208) 259584 ['block6g_se_excite[0][0]']
2D)
block6g_project_bn (BatchN (None, 2, 2, 208) 832 ['block6g_project_conv[0][0]']
ormalization)
block6g_drop (Dropout) (None, 2, 2, 208) 0 ['block6g_project_bn[0][0]']
block6g_add (Add) (None, 2, 2, 208) 0 ['block6g_drop[0][0]',
'block6f_add[0][0]']
block6h_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6g_add[0][0]']
D)
block6h_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6h_expand_conv[0][0]']
rmalization)
block6h_expand_activation (None, 2, 2, 1248) 0 ['block6h_expand_bn[0][0]']
(Activation)
block6h_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6h_expand_activation[0]
Conv2D) [0]']
block6h_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6h_dwconv2[0][0]']
tion)
block6h_activation (Activa (None, 2, 2, 1248) 0 ['block6h_bn[0][0]']
tion)
block6h_se_squeeze (Global (None, 1248) 0 ['block6h_activation[0][0]']
AveragePooling2D)
block6h_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6h_se_squeeze[0][0]']
e)
block6h_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6h_se_reshape[0][0]']
block6h_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6h_se_reduce[0][0]']
block6h_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6h_activation[0][0]',
y) 'block6h_se_expand[0][0]']
block6h_project_conv (Conv (None, 2, 2, 208) 259584 ['block6h_se_excite[0][0]']
2D)
block6h_project_bn (BatchN (None, 2, 2, 208) 832 ['block6h_project_conv[0][0]']
ormalization)
block6h_drop (Dropout) (None, 2, 2, 208) 0 ['block6h_project_bn[0][0]']
block6h_add (Add) (None, 2, 2, 208) 0 ['block6h_drop[0][0]',
'block6g_add[0][0]']
block6i_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6h_add[0][0]']
D)
block6i_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6i_expand_conv[0][0]']
rmalization)
block6i_expand_activation (None, 2, 2, 1248) 0 ['block6i_expand_bn[0][0]']
(Activation)
block6i_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6i_expand_activation[0]
Conv2D) [0]']
block6i_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6i_dwconv2[0][0]']
tion)
block6i_activation (Activa (None, 2, 2, 1248) 0 ['block6i_bn[0][0]']
tion)
block6i_se_squeeze (Global (None, 1248) 0 ['block6i_activation[0][0]']
AveragePooling2D)
block6i_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6i_se_squeeze[0][0]']
e)
block6i_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6i_se_reshape[0][0]']
block6i_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6i_se_reduce[0][0]']
block6i_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6i_activation[0][0]',
y) 'block6i_se_expand[0][0]']
block6i_project_conv (Conv (None, 2, 2, 208) 259584 ['block6i_se_excite[0][0]']
2D)
block6i_project_bn (BatchN (None, 2, 2, 208) 832 ['block6i_project_conv[0][0]']
ormalization)
block6i_drop (Dropout) (None, 2, 2, 208) 0 ['block6i_project_bn[0][0]']
block6i_add (Add) (None, 2, 2, 208) 0 ['block6i_drop[0][0]',
'block6h_add[0][0]']
block6j_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6i_add[0][0]']
D)
block6j_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6j_expand_conv[0][0]']
rmalization)
block6j_expand_activation (None, 2, 2, 1248) 0 ['block6j_expand_bn[0][0]']
(Activation)
block6j_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6j_expand_activation[0]
Conv2D) [0]']
block6j_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6j_dwconv2[0][0]']
tion)
block6j_activation (Activa (None, 2, 2, 1248) 0 ['block6j_bn[0][0]']
tion)
block6j_se_squeeze (Global (None, 1248) 0 ['block6j_activation[0][0]']
AveragePooling2D)
block6j_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6j_se_squeeze[0][0]']
e)
block6j_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6j_se_reshape[0][0]']
block6j_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6j_se_reduce[0][0]']
block6j_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6j_activation[0][0]',
y) 'block6j_se_expand[0][0]']
block6j_project_conv (Conv (None, 2, 2, 208) 259584 ['block6j_se_excite[0][0]']
2D)
block6j_project_bn (BatchN (None, 2, 2, 208) 832 ['block6j_project_conv[0][0]']
ormalization)
block6j_drop (Dropout) (None, 2, 2, 208) 0 ['block6j_project_bn[0][0]']
block6j_add (Add) (None, 2, 2, 208) 0 ['block6j_drop[0][0]',
'block6i_add[0][0]']
top_conv (Conv2D) (None, 2, 2, 1408) 292864 ['block6j_add[0][0]']
top_bn (BatchNormalization (None, 2, 2, 1408) 5632 ['top_conv[0][0]']
)
top_activation (Activation (None, 2, 2, 1408) 0 ['top_bn[0][0]']
)
==================================================================================================
Total params: 8769374 (33.45 MB)
Trainable params: 8687086 (33.14 MB)
Non-trainable params: 82288 (321.44 KB)
__________________________________________________________________________________________________
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=False,
fill_mode='nearest'
)
# Define the image size and paths
image_size = 48
train_path = "Facial_emotion_images/train"
test_path = "Facial_emotion_images/test"
validation_path = "Facial_emotion_images/validation"
# Initialize the data generator with rescaling
datagen = ImageDataGenerator(rescale=1./255)
# Prepare the data generators
train_generator = datagen.flow_from_directory(
train_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical')
validation_generator = datagen.flow_from_directory(
validation_path,
target_size=(48, 48),
batch_size=32, # Keep the same batch size for simplicity
class_mode='categorical')
test_generator = datagen.flow_from_directory(
test_path,
target_size=(48, 48),
shuffle=False)
Found 15109 images belonging to 4 classes. Found 4977 images belonging to 4 classes. Found 128 images belonging to 4 classes.
# Load EfficientNetB0 model pre-trained on ImageNet without the top layer
base_model = EfficientNetB0(weights='imagenet', include_top=False, input_shape=(48, 48, 3))
# Add custom layers on top of the base model
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
x = Dropout(0.2)(x)
x = Dense(1024, activation='relu')(x)
x = Dropout(0.2)(x) # Adjusted intermediate layer
predictions = Dense(4, activation='softmax')(x) # Adjust the number of units to match the number of classes
# Define the new model
efficient_model2 = Model(inputs=base_model.input, outputs=predictions)
efficient_model2.compile(optimizer=AdamW(learning_rate=0.000025), loss='categorical_crossentropy', metrics=['accuracy'])
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, min_lr=0.000001)
model_checkpoint = ModelCheckpoint('best_model.h5', monitor='val_accuracy', save_best_only=True)
early_stopping = EarlyStopping(monitor='val_loss', patience=7, restore_best_weights=True)
history = efficient_model2.fit(
train_generator,
steps_per_epoch=train_generator.samples // train_generator.batch_size,
epochs=50,
validation_data=validation_generator,
validation_steps=validation_generator.samples // validation_generator.batch_size,
class_weight=class_weights_adjusted,
callbacks=[reduce_lr, model_checkpoint, early_stopping]
)
Epoch 1/50 471/472 [============================>.] - ETA: 0s - loss: 1.3364 - accuracy: 0.3444
/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py:3103: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
saving_api.save_model(
472/472 [==============================] - 61s 56ms/step - loss: 1.3362 - accuracy: 0.3445 - val_loss: 1.4205 - val_accuracy: 0.2494 - lr: 2.5000e-05 Epoch 2/50 472/472 [==============================] - 24s 51ms/step - loss: 1.1789 - accuracy: 0.4744 - val_loss: 1.5377 - val_accuracy: 0.2607 - lr: 2.5000e-05 Epoch 3/50 472/472 [==============================] - 24s 50ms/step - loss: 1.0906 - accuracy: 0.5233 - val_loss: 1.3381 - val_accuracy: 0.3417 - lr: 2.5000e-05 Epoch 4/50 472/472 [==============================] - 24s 51ms/step - loss: 1.0268 - accuracy: 0.5559 - val_loss: 1.3126 - val_accuracy: 0.3831 - lr: 2.5000e-05 Epoch 5/50 472/472 [==============================] - 23s 50ms/step - loss: 0.9674 - accuracy: 0.5873 - val_loss: 1.4013 - val_accuracy: 0.3456 - lr: 2.5000e-05 Epoch 6/50 472/472 [==============================] - 24s 52ms/step - loss: 0.9249 - accuracy: 0.6089 - val_loss: 1.3047 - val_accuracy: 0.4036 - lr: 2.5000e-05 Epoch 7/50 472/472 [==============================] - 24s 51ms/step - loss: 0.8887 - accuracy: 0.6251 - val_loss: 1.2046 - val_accuracy: 0.4643 - lr: 2.5000e-05 Epoch 8/50 472/472 [==============================] - 23s 49ms/step - loss: 0.8461 - accuracy: 0.6501 - val_loss: 1.2797 - val_accuracy: 0.4194 - lr: 2.5000e-05 Epoch 9/50 472/472 [==============================] - 23s 50ms/step - loss: 0.8105 - accuracy: 0.6625 - val_loss: 1.6911 - val_accuracy: 0.2988 - lr: 2.5000e-05 Epoch 10/50 472/472 [==============================] - 23s 49ms/step - loss: 0.7762 - accuracy: 0.6786 - val_loss: 1.5159 - val_accuracy: 0.3510 - lr: 2.5000e-05 Epoch 11/50 472/472 [==============================] - 23s 49ms/step - loss: 0.7396 - accuracy: 0.6985 - val_loss: 1.3115 - val_accuracy: 0.4635 - lr: 2.5000e-05 Epoch 12/50 472/472 [==============================] - 23s 49ms/step - loss: 0.7042 - accuracy: 0.7141 - val_loss: 1.4750 - val_accuracy: 0.4284 - lr: 2.5000e-05 Epoch 13/50 472/472 [==============================] - 24s 51ms/step - loss: 0.6657 - accuracy: 0.7307 - val_loss: 1.2258 - val_accuracy: 0.4984 - lr: 5.0000e-06 Epoch 14/50 472/472 [==============================] - 24s 51ms/step - loss: 0.6549 - accuracy: 0.7325 - val_loss: 1.1271 - val_accuracy: 0.5583 - lr: 5.0000e-06 Epoch 15/50 472/472 [==============================] - 24s 51ms/step - loss: 0.6461 - accuracy: 0.7392 - val_loss: 1.0258 - val_accuracy: 0.5927 - lr: 5.0000e-06 Epoch 16/50 472/472 [==============================] - 24s 51ms/step - loss: 0.6434 - accuracy: 0.7391 - val_loss: 0.9484 - val_accuracy: 0.6177 - lr: 5.0000e-06 Epoch 17/50 472/472 [==============================] - 23s 50ms/step - loss: 0.6222 - accuracy: 0.7500 - val_loss: 1.1752 - val_accuracy: 0.5224 - lr: 5.0000e-06 Epoch 18/50 472/472 [==============================] - 23s 49ms/step - loss: 0.6238 - accuracy: 0.7521 - val_loss: 1.0146 - val_accuracy: 0.6067 - lr: 5.0000e-06 Epoch 19/50 472/472 [==============================] - 24s 50ms/step - loss: 0.6201 - accuracy: 0.7529 - val_loss: 1.0254 - val_accuracy: 0.6173 - lr: 5.0000e-06 Epoch 20/50 472/472 [==============================] - 24s 51ms/step - loss: 0.6098 - accuracy: 0.7553 - val_loss: 0.9382 - val_accuracy: 0.6367 - lr: 5.0000e-06 Epoch 21/50 472/472 [==============================] - 23s 49ms/step - loss: 0.6099 - accuracy: 0.7577 - val_loss: 1.0727 - val_accuracy: 0.5831 - lr: 5.0000e-06 Epoch 22/50 472/472 [==============================] - 24s 50ms/step - loss: 0.5948 - accuracy: 0.7632 - val_loss: 0.9776 - val_accuracy: 0.6135 - lr: 5.0000e-06 Epoch 23/50 472/472 [==============================] - 23s 49ms/step - loss: 0.5924 - accuracy: 0.7661 - val_loss: 1.0028 - val_accuracy: 0.6190 - lr: 5.0000e-06 Epoch 24/50 472/472 [==============================] - 23s 49ms/step - loss: 0.5832 - accuracy: 0.7723 - val_loss: 1.0490 - val_accuracy: 0.6113 - lr: 5.0000e-06 Epoch 25/50 472/472 [==============================] - 24s 51ms/step - loss: 0.5757 - accuracy: 0.7692 - val_loss: 0.9597 - val_accuracy: 0.6375 - lr: 5.0000e-06 Epoch 26/50 472/472 [==============================] - 24s 50ms/step - loss: 0.5759 - accuracy: 0.7736 - val_loss: 0.9805 - val_accuracy: 0.6216 - lr: 1.0000e-06 Epoch 27/50 472/472 [==============================] - 23s 50ms/step - loss: 0.5630 - accuracy: 0.7769 - val_loss: 0.9584 - val_accuracy: 0.6250 - lr: 1.0000e-06
# Evaluate the model on the test set
val_loss, val_accuracy = efficient_model2.evaluate(
test_generator,
steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
4/4 [==============================] - 0s 17ms/step - loss: 0.8464 - accuracy: 0.6484 Test Loss: 0.84635990858078 Test Accuracy: 0.6484375
plot_model_performance_and_confusion_matrix(efficient_model2, history, test_generator,
metrics_function=lambda true, pred: print(classification_report(true, pred)))
from IPython.display import HTML, display
# Assuming val_loss and val_accuracy are defined from your model's evaluation
# Evaluate the model on the test set
val_loss, val_accuracy = efficient_model2.evaluate(
test_generator,
steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
# Format the evaluation results as HTML
output_html = f"""
<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>
<p style='color: red;'><b>Test Loss:</b> efficient_model2 </p>
<p style='color: red;'><b>Test Loss:</b> {val_loss:.2f}</p>
<p style='color: blue;'><b>Test Accuracy:</b> {val_accuracy:.2%} </p>
</div>
"""
# Display the styled output
display(HTML(output_html))
4/4 [==============================] - 0s 15ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
precision recall f1-score support
0 0.67 0.81 0.73 32
1 0.57 0.72 0.64 32
2 0.54 0.44 0.48 32
3 0.87 0.62 0.73 32
accuracy 0.65 128
macro avg 0.66 0.65 0.65 128
weighted avg 0.66 0.65 0.65 128
4/4 [==============================] - 0s 21ms/step - loss: 0.8464 - accuracy: 0.6484 Test Loss: 0.84635990858078 Test Accuracy: 0.6484375
Test Loss: efficient_model2
Test Loss: 0.85
Test Accuracy: 64.84%
Observations and Insights:__
Note: You can even go back and build your own architecture on top of the VGG16 Transfer layer and see if you can improve the performance.
Think About It:
The EfficientNetB0-based model tailored for a 4-class emotion classification task exhibits an overall accuracy of 62%. Performance metrics indicate varied effectiveness across classes:
Happy class shows moderate precision and slightly higher recall, achieving a balance reflected in an F1-score of 0.70. Neutral and Sad classes have lower precision and recall, resulting in F1-scores of 0.56 and 0.56, respectively. This suggests challenges in distinguishing these emotions accurately. Surprise class demonstrates higher precision but reduced recall, with an F1-score of 0.67, indicating better identification but room for improved coverage. These results suggest that while the model can identify certain emotions with reasonable accuracy, its performance is inconsistent across all classes. The lower overall accuracy and mixed precision and recall scores highlight areas for improvement, particularly in enhancing the model's ability to recognize 'neutral' and 'sad' emotions more accurately. Optimization strategies might include adjusting the model's architecture, fine-tuning the learning rate, and employing targeted data augmentation to address these disparities.
:Observations and Insights:__
To evaluate the overall performance of the transfer learning architectures mentioned (VGG16, ResNet50, and EfficientNetB0) for the emotion classification task, let's compare their reported metrics:
VGG16-based Model:
Accuracy: 77% Best performing class: 'Surprise' with high precision and recall. Challenges: Lower precision in 'neutral' and lower recall in 'sad'. ResNet50-based Model:
Accuracy: 75% Best performing class: 'Happy' and 'Surprise' with high precision and recall. Challenges: Lower precision for 'neutral' and 'sad', indicating difficulty in distinguishing these emotions accurately. EfficientNetB0-based Model:
Accuracy: 62% Shows moderate to good performance in recognizing 'happy' and 'surprise' but struggles with 'neutral' and 'sad'. The lower overall accuracy compared to VGG16 and ResNet50.
The VGG16 and ResNet50 models have shown similar performance levels, with both achieving mid to high 70s in accuracy. They are relatively close in their ability to classify emotions, with strengths in identifying 'happy' and 'surprise' emotions accurately. The EfficientNetB0 model lagged behind in performance, with an accuracy of 62%, indicating that it might not have been as effective for this particular task or dataset. While accuracies are decent for VGG16 and ResNet50,there is still alot of room for improvement.
The task of computer vision is i
CONVOLUTIONAL NETWORK #2
plot_model_performance_and_confusion_matrix(cnn_model_updated, history, test_generator,
metrics_function=lambda true, pred: print(classification_report(true, pred)))
4/4 [==============================] - 1s 11ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
precision recall f1-score support
0 0.82 0.84 0.83 32
1 0.72 0.81 0.76 32
2 0.79 0.69 0.73 32
3 0.97 0.94 0.95 32
accuracy 0.82 128
macro avg 0.82 0.82 0.82 128
weighted avg 0.82 0.82 0.82 128
Now that we have tried multiple pre-trained models, let's build a complex CNN architecture and see if we can get better performance.
In this section, we will build a more complex Convolutional Neural Network Model that has close to as many parameters as we had in our Transfer Learning Models. However, we will have only 1 input channel for our input images.
In this section, we are creating data loaders which we will use as inputs to the more Complicated Convolutional Neural Network. We will go ahead with color_mode = 'grayscale'.
# Set the random Seed
np.random.seed(42)
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=False,
fill_mode='nearest'
)
# Define the image size and paths
image_size = 48
train_path = "Facial_emotion_images/train"
test_path = "Facial_emotion_images/test"
validation_path = "Facial_emotion_images/validation"
# Initialize the data generator with rescaling
datagen = ImageDataGenerator(rescale=1./255)
# Prepare the data generators
train_generator = datagen.flow_from_directory(
train_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical')
validation_generator = datagen.flow_from_directory(
validation_path,
target_size=(48, 48),
batch_size=32, # Keep the same batch size for simplicity
class_mode='categorical')
test_generator = datagen.flow_from_directory(
test_path,
target_size=(48, 48),
shuffle=False)
Found 15109 images belonging to 4 classes. Found 4977 images belonging to 4 classes. Found 128 images belonging to 4 classes.
from tensorflow.keras.preprocessing.image import ImageDataGenerator
# Create more augmented images for 'sad' and 'neutral' classes
augmentation_for_sad_neutral = ImageDataGenerator(
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.15,
zoom_range=0.15,
horizontal_flip=True,
fill_mode='nearest'
)
cnn_model_4 = Sequential([
Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(48, 48, 3)),
BatchNormalization(),
Conv2D(64, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.3), # Slightly reduced dropout
# Second Convolutional Block
Conv2D(128, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(256, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.3), # Adjusted dropout
# Third Convolutional Block
Conv2D(256, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(512, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.4), # Adjusted dropout
# Fourth Convolutional Block
Conv2D(1024, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.2), # Keeping consistent dropout
# Convolutional Block 5
Conv2D(1024, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.2), # Keeping consistent dropout
# Global Average Pooling replaces Flatten to reduce parameters and potential overfitting
GlobalAveragePooling2D(),
Flatten(),
# Dense Layers
Dense(128, activation='relu'),
Dense(512, activation='relu'),
Dropout(0.2),
Dense(1024, activation='relu'),
Dense(2048, activation='relu'),
Dropout(0.2),
Dense(128, activation='relu'),
Dropout(0.2),
Dense(4, activation='softmax')
])
cnn_model_4.summary() # Print the model summary
Model: "sequential_47"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_218 (Conv2D) (None, 48, 48, 64) 1792
batch_normalization_154 (B (None, 48, 48, 64) 256
atchNormalization)
conv2d_219 (Conv2D) (None, 48, 48, 64) 36928
max_pooling2d_128 (MaxPool (None, 24, 24, 64) 0
ing2D)
dropout_186 (Dropout) (None, 24, 24, 64) 0
conv2d_220 (Conv2D) (None, 24, 24, 128) 73856
batch_normalization_155 (B (None, 24, 24, 128) 512
atchNormalization)
conv2d_221 (Conv2D) (None, 24, 24, 256) 295168
max_pooling2d_129 (MaxPool (None, 12, 12, 256) 0
ing2D)
dropout_187 (Dropout) (None, 12, 12, 256) 0
conv2d_222 (Conv2D) (None, 12, 12, 256) 590080
batch_normalization_156 (B (None, 12, 12, 256) 1024
atchNormalization)
conv2d_223 (Conv2D) (None, 12, 12, 512) 1180160
batch_normalization_157 (B (None, 12, 12, 512) 2048
atchNormalization)
max_pooling2d_130 (MaxPool (None, 6, 6, 512) 0
ing2D)
dropout_188 (Dropout) (None, 6, 6, 512) 0
conv2d_224 (Conv2D) (None, 6, 6, 1024) 4719616
batch_normalization_158 (B (None, 6, 6, 1024) 4096
atchNormalization)
max_pooling2d_131 (MaxPool (None, 3, 3, 1024) 0
ing2D)
dropout_189 (Dropout) (None, 3, 3, 1024) 0
conv2d_225 (Conv2D) (None, 3, 3, 1024) 9438208
batch_normalization_159 (B (None, 3, 3, 1024) 4096
atchNormalization)
max_pooling2d_132 (MaxPool (None, 1, 1, 1024) 0
ing2D)
dropout_190 (Dropout) (None, 1, 1, 1024) 0
global_average_pooling2d_3 (None, 1024) 0
1 (GlobalAveragePooling2D)
flatten_34 (Flatten) (None, 1024) 0
dense_158 (Dense) (None, 128) 131200
dense_159 (Dense) (None, 512) 66048
dropout_191 (Dropout) (None, 512) 0
dense_160 (Dense) (None, 1024) 525312
dense_161 (Dense) (None, 2048) 2099200
dropout_192 (Dropout) (None, 2048) 0
dense_162 (Dense) (None, 128) 262272
dropout_193 (Dropout) (None, 128) 0
dense_163 (Dense) (None, 4) 516
=================================================================
Total params: 19432388 (74.13 MB)
Trainable params: 19426372 (74.11 MB)
Non-trainable params: 6016 (23.50 KB)
_________________________________________________________________
cnn_model_4.compile(optimizer=AdamW(learning_rate=0.00003),
loss='categorical_crossentropy',
metrics=['accuracy'])
early_stopping = EarlyStopping(monitor='val_loss', patience=10, verbose=1, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, verbose=1, min_lr=0.00001)
history = cnn_model_4.fit(
train_generator,
epochs=30,
verbose=1,
callbacks=[early_stopping, reduce_lr],
validation_data=validation_generator,
class_weight=class_weights_adjusted
)
Epoch 1/30 473/473 [==============================] - 33s 57ms/step - loss: 1.3675 - accuracy: 0.2974 - val_loss: 1.4441 - val_accuracy: 0.2465 - lr: 3.0000e-05 Epoch 2/30 473/473 [==============================] - 27s 57ms/step - loss: 1.3172 - accuracy: 0.3478 - val_loss: 1.4579 - val_accuracy: 0.2769 - lr: 3.0000e-05 Epoch 3/30 473/473 [==============================] - 28s 58ms/step - loss: 1.2367 - accuracy: 0.4090 - val_loss: 1.1947 - val_accuracy: 0.4621 - lr: 3.0000e-05 Epoch 4/30 473/473 [==============================] - 28s 59ms/step - loss: 1.1550 - accuracy: 0.4691 - val_loss: 1.0658 - val_accuracy: 0.5182 - lr: 3.0000e-05 Epoch 5/30 473/473 [==============================] - 28s 59ms/step - loss: 1.0598 - accuracy: 0.5358 - val_loss: 1.0690 - val_accuracy: 0.5339 - lr: 3.0000e-05 Epoch 6/30 473/473 [==============================] - 28s 59ms/step - loss: 0.9991 - accuracy: 0.5732 - val_loss: 0.9468 - val_accuracy: 0.6020 - lr: 3.0000e-05 Epoch 7/30 473/473 [==============================] - 28s 59ms/step - loss: 0.9407 - accuracy: 0.5997 - val_loss: 0.9827 - val_accuracy: 0.5990 - lr: 3.0000e-05 Epoch 8/30 473/473 [==============================] - 28s 59ms/step - loss: 0.8964 - accuracy: 0.6178 - val_loss: 0.9243 - val_accuracy: 0.6401 - lr: 3.0000e-05 Epoch 9/30 473/473 [==============================] - 28s 59ms/step - loss: 0.8639 - accuracy: 0.6409 - val_loss: 0.8495 - val_accuracy: 0.6703 - lr: 3.0000e-05 Epoch 10/30 473/473 [==============================] - 28s 59ms/step - loss: 0.8245 - accuracy: 0.6535 - val_loss: 0.8480 - val_accuracy: 0.6773 - lr: 3.0000e-05 Epoch 11/30 473/473 [==============================] - 28s 59ms/step - loss: 0.8008 - accuracy: 0.6653 - val_loss: 0.8084 - val_accuracy: 0.6900 - lr: 3.0000e-05 Epoch 12/30 473/473 [==============================] - 28s 59ms/step - loss: 0.7659 - accuracy: 0.6814 - val_loss: 0.7696 - val_accuracy: 0.7085 - lr: 3.0000e-05 Epoch 13/30 473/473 [==============================] - 28s 59ms/step - loss: 0.7424 - accuracy: 0.6950 - val_loss: 0.7819 - val_accuracy: 0.7012 - lr: 3.0000e-05 Epoch 14/30 473/473 [==============================] - 28s 59ms/step - loss: 0.7101 - accuracy: 0.7108 - val_loss: 0.7668 - val_accuracy: 0.6992 - lr: 3.0000e-05 Epoch 15/30 473/473 [==============================] - 28s 59ms/step - loss: 0.6990 - accuracy: 0.7163 - val_loss: 0.7166 - val_accuracy: 0.7219 - lr: 3.0000e-05 Epoch 16/30 473/473 [==============================] - 28s 59ms/step - loss: 0.6658 - accuracy: 0.7249 - val_loss: 0.6994 - val_accuracy: 0.7267 - lr: 3.0000e-05 Epoch 17/30 473/473 [==============================] - 28s 59ms/step - loss: 0.6449 - accuracy: 0.7385 - val_loss: 0.7149 - val_accuracy: 0.7312 - lr: 3.0000e-05 Epoch 18/30 473/473 [==============================] - 28s 59ms/step - loss: 0.6245 - accuracy: 0.7470 - val_loss: 0.7161 - val_accuracy: 0.7346 - lr: 3.0000e-05 Epoch 19/30 473/473 [==============================] - 28s 59ms/step - loss: 0.6038 - accuracy: 0.7531 - val_loss: 0.6973 - val_accuracy: 0.7378 - lr: 3.0000e-05 Epoch 20/30 473/473 [==============================] - 28s 59ms/step - loss: 0.5787 - accuracy: 0.7652 - val_loss: 0.7712 - val_accuracy: 0.7233 - lr: 3.0000e-05 Epoch 21/30 473/473 [==============================] - 28s 59ms/step - loss: 0.5627 - accuracy: 0.7734 - val_loss: 0.6959 - val_accuracy: 0.7380 - lr: 3.0000e-05 Epoch 22/30 473/473 [==============================] - 28s 59ms/step - loss: 0.5365 - accuracy: 0.7804 - val_loss: 0.7029 - val_accuracy: 0.7408 - lr: 3.0000e-05 Epoch 23/30 473/473 [==============================] - 28s 59ms/step - loss: 0.5206 - accuracy: 0.7927 - val_loss: 0.7007 - val_accuracy: 0.7454 - lr: 3.0000e-05 Epoch 24/30 473/473 [==============================] - 28s 59ms/step - loss: 0.4909 - accuracy: 0.8044 - val_loss: 0.7170 - val_accuracy: 0.7478 - lr: 3.0000e-05 Epoch 25/30 473/473 [==============================] - 28s 59ms/step - loss: 0.4651 - accuracy: 0.8154 - val_loss: 0.7634 - val_accuracy: 0.7370 - lr: 3.0000e-05 Epoch 26/30 473/473 [==============================] - ETA: 0s - loss: 0.4524 - accuracy: 0.8224 Epoch 26: ReduceLROnPlateau reducing learning rate to 1e-05. 473/473 [==============================] - 28s 59ms/step - loss: 0.4524 - accuracy: 0.8224 - val_loss: 0.7263 - val_accuracy: 0.7458 - lr: 3.0000e-05 Epoch 27/30 473/473 [==============================] - 28s 59ms/step - loss: 0.3850 - accuracy: 0.8508 - val_loss: 0.7633 - val_accuracy: 0.7456 - lr: 1.0000e-05 Epoch 28/30 473/473 [==============================] - 28s 59ms/step - loss: 0.3626 - accuracy: 0.8564 - val_loss: 0.7651 - val_accuracy: 0.7468 - lr: 1.0000e-05 Epoch 29/30 473/473 [==============================] - 28s 59ms/step - loss: 0.3502 - accuracy: 0.8602 - val_loss: 0.7947 - val_accuracy: 0.7454 - lr: 1.0000e-05 Epoch 30/30 473/473 [==============================] - 28s 59ms/step - loss: 0.3374 - accuracy: 0.8695 - val_loss: 0.8206 - val_accuracy: 0.7430 - lr: 1.0000e-05
history = cnn_model_4.fit(
train_generator,
epochs=75,
verbose=1,
callbacks=[reduce_lr],
validation_data=validation_generator,
class_weight=class_weights_adjusted
)
Epoch 1/75 473/473 [==============================] - 32s 57ms/step - loss: 1.3529 - accuracy: 0.3203 - val_loss: 1.3962 - val_accuracy: 0.3649 - lr: 3.0000e-05 Epoch 2/75 473/473 [==============================] - 27s 57ms/step - loss: 1.3064 - accuracy: 0.3630 - val_loss: 1.3467 - val_accuracy: 0.3757 - lr: 3.0000e-05 Epoch 3/75 473/473 [==============================] - 27s 58ms/step - loss: 1.2189 - accuracy: 0.4201 - val_loss: 1.1732 - val_accuracy: 0.4531 - lr: 3.0000e-05 Epoch 4/75 473/473 [==============================] - 28s 58ms/step - loss: 1.1095 - accuracy: 0.5029 - val_loss: 1.1512 - val_accuracy: 0.5142 - lr: 3.0000e-05 Epoch 5/75 473/473 [==============================] - 28s 59ms/step - loss: 1.0259 - accuracy: 0.5528 - val_loss: 1.0352 - val_accuracy: 0.5738 - lr: 3.0000e-05 Epoch 6/75 473/473 [==============================] - 28s 58ms/step - loss: 0.9608 - accuracy: 0.5862 - val_loss: 1.0371 - val_accuracy: 0.5799 - lr: 3.0000e-05 Epoch 7/75 473/473 [==============================] - 28s 58ms/step - loss: 0.9163 - accuracy: 0.6108 - val_loss: 0.9709 - val_accuracy: 0.6180 - lr: 3.0000e-05 Epoch 8/75 473/473 [==============================] - 28s 58ms/step - loss: 0.8684 - accuracy: 0.6329 - val_loss: 0.9718 - val_accuracy: 0.6339 - lr: 3.0000e-05 Epoch 9/75 473/473 [==============================] - 28s 58ms/step - loss: 0.8387 - accuracy: 0.6496 - val_loss: 0.9744 - val_accuracy: 0.6349 - lr: 3.0000e-05 Epoch 10/75 473/473 [==============================] - 28s 58ms/step - loss: 0.8003 - accuracy: 0.6652 - val_loss: 0.8822 - val_accuracy: 0.6787 - lr: 3.0000e-05 Epoch 11/75 473/473 [==============================] - 28s 59ms/step - loss: 0.7792 - accuracy: 0.6750 - val_loss: 0.8308 - val_accuracy: 0.6751 - lr: 3.0000e-05 Epoch 12/75 473/473 [==============================] - 28s 58ms/step - loss: 0.7439 - accuracy: 0.6922 - val_loss: 0.7526 - val_accuracy: 0.7028 - lr: 3.0000e-05 Epoch 13/75 473/473 [==============================] - 28s 58ms/step - loss: 0.7282 - accuracy: 0.6992 - val_loss: 0.7748 - val_accuracy: 0.6801 - lr: 3.0000e-05 Epoch 14/75 473/473 [==============================] - 28s 59ms/step - loss: 0.7001 - accuracy: 0.7132 - val_loss: 0.7531 - val_accuracy: 0.7030 - lr: 3.0000e-05 Epoch 15/75 473/473 [==============================] - 28s 58ms/step - loss: 0.6800 - accuracy: 0.7193 - val_loss: 0.7626 - val_accuracy: 0.7213 - lr: 3.0000e-05 Epoch 16/75 473/473 [==============================] - 28s 59ms/step - loss: 0.6534 - accuracy: 0.7336 - val_loss: 0.7654 - val_accuracy: 0.7189 - lr: 3.0000e-05 Epoch 17/75 473/473 [==============================] - 28s 58ms/step - loss: 0.6306 - accuracy: 0.7451 - val_loss: 0.7242 - val_accuracy: 0.7286 - lr: 3.0000e-05 Epoch 18/75 473/473 [==============================] - 28s 58ms/step - loss: 0.6145 - accuracy: 0.7500 - val_loss: 0.7546 - val_accuracy: 0.7091 - lr: 3.0000e-05 Epoch 19/75 473/473 [==============================] - 28s 58ms/step - loss: 0.5924 - accuracy: 0.7586 - val_loss: 0.6986 - val_accuracy: 0.7360 - lr: 3.0000e-05 Epoch 20/75 473/473 [==============================] - 28s 58ms/step - loss: 0.5708 - accuracy: 0.7662 - val_loss: 0.7054 - val_accuracy: 0.7362 - lr: 3.0000e-05 Epoch 21/75 473/473 [==============================] - 28s 58ms/step - loss: 0.5444 - accuracy: 0.7811 - val_loss: 0.7600 - val_accuracy: 0.7330 - lr: 3.0000e-05 Epoch 22/75 473/473 [==============================] - 28s 59ms/step - loss: 0.5204 - accuracy: 0.7886 - val_loss: 0.7341 - val_accuracy: 0.7334 - lr: 3.0000e-05 Epoch 23/75 473/473 [==============================] - 28s 58ms/step - loss: 0.4989 - accuracy: 0.7985 - val_loss: 0.7955 - val_accuracy: 0.7273 - lr: 3.0000e-05 Epoch 24/75 473/473 [==============================] - ETA: 0s - loss: 0.4749 - accuracy: 0.8068 Epoch 24: ReduceLROnPlateau reducing learning rate to 1e-05. 473/473 [==============================] - 28s 58ms/step - loss: 0.4749 - accuracy: 0.8068 - val_loss: 0.7177 - val_accuracy: 0.7454 - lr: 3.0000e-05 Epoch 25/75 473/473 [==============================] - 28s 58ms/step - loss: 0.4228 - accuracy: 0.8315 - val_loss: 0.7474 - val_accuracy: 0.7470 - lr: 1.0000e-05 Epoch 26/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3946 - accuracy: 0.8453 - val_loss: 0.7789 - val_accuracy: 0.7432 - lr: 1.0000e-05 Epoch 27/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3869 - accuracy: 0.8459 - val_loss: 0.7746 - val_accuracy: 0.7444 - lr: 1.0000e-05 Epoch 28/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3722 - accuracy: 0.8547 - val_loss: 0.7694 - val_accuracy: 0.7559 - lr: 1.0000e-05 Epoch 29/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3528 - accuracy: 0.8617 - val_loss: 0.8082 - val_accuracy: 0.7454 - lr: 1.0000e-05 Epoch 30/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3439 - accuracy: 0.8594 - val_loss: 0.7955 - val_accuracy: 0.7480 - lr: 1.0000e-05 Epoch 31/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3311 - accuracy: 0.8717 - val_loss: 0.8314 - val_accuracy: 0.7464 - lr: 1.0000e-05 Epoch 32/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3227 - accuracy: 0.8716 - val_loss: 0.8423 - val_accuracy: 0.7406 - lr: 1.0000e-05 Epoch 33/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3145 - accuracy: 0.8768 - val_loss: 0.8531 - val_accuracy: 0.7446 - lr: 1.0000e-05 Epoch 34/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3001 - accuracy: 0.8834 - val_loss: 0.8843 - val_accuracy: 0.7482 - lr: 1.0000e-05 Epoch 35/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2829 - accuracy: 0.8889 - val_loss: 0.9057 - val_accuracy: 0.7402 - lr: 1.0000e-05 Epoch 36/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2840 - accuracy: 0.8887 - val_loss: 0.8644 - val_accuracy: 0.7440 - lr: 1.0000e-05 Epoch 37/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2776 - accuracy: 0.8927 - val_loss: 0.8776 - val_accuracy: 0.7388 - lr: 1.0000e-05 Epoch 38/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2650 - accuracy: 0.8952 - val_loss: 0.9019 - val_accuracy: 0.7422 - lr: 1.0000e-05 Epoch 39/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2612 - accuracy: 0.8976 - val_loss: 0.9259 - val_accuracy: 0.7448 - lr: 1.0000e-05 Epoch 40/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2482 - accuracy: 0.9060 - val_loss: 0.9300 - val_accuracy: 0.7476 - lr: 1.0000e-05 Epoch 41/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2518 - accuracy: 0.9027 - val_loss: 0.8982 - val_accuracy: 0.7476 - lr: 1.0000e-05 Epoch 42/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2349 - accuracy: 0.9086 - val_loss: 0.9488 - val_accuracy: 0.7470 - lr: 1.0000e-05 Epoch 43/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2288 - accuracy: 0.9130 - val_loss: 0.9425 - val_accuracy: 0.7519 - lr: 1.0000e-05 Epoch 44/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2199 - accuracy: 0.9153 - val_loss: 0.9910 - val_accuracy: 0.7416 - lr: 1.0000e-05 Epoch 45/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2143 - accuracy: 0.9180 - val_loss: 0.9927 - val_accuracy: 0.7480 - lr: 1.0000e-05 Epoch 46/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2127 - accuracy: 0.9192 - val_loss: 0.9895 - val_accuracy: 0.7498 - lr: 1.0000e-05 Epoch 47/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2106 - accuracy: 0.9196 - val_loss: 1.0095 - val_accuracy: 0.7438 - lr: 1.0000e-05 Epoch 48/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1981 - accuracy: 0.9259 - val_loss: 1.0120 - val_accuracy: 0.7505 - lr: 1.0000e-05 Epoch 49/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1980 - accuracy: 0.9231 - val_loss: 0.9901 - val_accuracy: 0.7490 - lr: 1.0000e-05 Epoch 50/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1898 - accuracy: 0.9265 - val_loss: 1.0584 - val_accuracy: 0.7418 - lr: 1.0000e-05 Epoch 51/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1845 - accuracy: 0.9279 - val_loss: 1.0573 - val_accuracy: 0.7533 - lr: 1.0000e-05 Epoch 52/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1808 - accuracy: 0.9321 - val_loss: 1.0527 - val_accuracy: 0.7472 - lr: 1.0000e-05 Epoch 53/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1741 - accuracy: 0.9338 - val_loss: 1.0805 - val_accuracy: 0.7430 - lr: 1.0000e-05 Epoch 54/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1675 - accuracy: 0.9385 - val_loss: 1.0889 - val_accuracy: 0.7458 - lr: 1.0000e-05 Epoch 55/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1632 - accuracy: 0.9375 - val_loss: 1.0601 - val_accuracy: 0.7454 - lr: 1.0000e-05 Epoch 56/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1649 - accuracy: 0.9381 - val_loss: 1.0953 - val_accuracy: 0.7438 - lr: 1.0000e-05 Epoch 57/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1606 - accuracy: 0.9389 - val_loss: 1.0946 - val_accuracy: 0.7482 - lr: 1.0000e-05 Epoch 58/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1558 - accuracy: 0.9386 - val_loss: 1.1140 - val_accuracy: 0.7452 - lr: 1.0000e-05 Epoch 59/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1515 - accuracy: 0.9437 - val_loss: 1.1096 - val_accuracy: 0.7472 - lr: 1.0000e-05 Epoch 60/75 473/473 [==============================] - 28s 59ms/step - loss: 0.1411 - accuracy: 0.9466 - val_loss: 1.1481 - val_accuracy: 0.7390 - lr: 1.0000e-05 Epoch 61/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1419 - accuracy: 0.9473 - val_loss: 1.1623 - val_accuracy: 0.7410 - lr: 1.0000e-05 Epoch 62/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1389 - accuracy: 0.9478 - val_loss: 1.1749 - val_accuracy: 0.7426 - lr: 1.0000e-05 Epoch 63/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1333 - accuracy: 0.9508 - val_loss: 1.1665 - val_accuracy: 0.7472 - lr: 1.0000e-05 Epoch 64/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1287 - accuracy: 0.9514 - val_loss: 1.1912 - val_accuracy: 0.7503 - lr: 1.0000e-05 Epoch 65/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1321 - accuracy: 0.9507 - val_loss: 1.1520 - val_accuracy: 0.7386 - lr: 1.0000e-05 Epoch 66/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1255 - accuracy: 0.9532 - val_loss: 1.1690 - val_accuracy: 0.7535 - lr: 1.0000e-05 Epoch 67/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1196 - accuracy: 0.9538 - val_loss: 1.1468 - val_accuracy: 0.7527 - lr: 1.0000e-05 Epoch 68/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1157 - accuracy: 0.9570 - val_loss: 1.2080 - val_accuracy: 0.7444 - lr: 1.0000e-05 Epoch 69/75 473/473 [==============================] - 28s 59ms/step - loss: 0.1148 - accuracy: 0.9577 - val_loss: 1.2205 - val_accuracy: 0.7420 - lr: 1.0000e-05 Epoch 70/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1019 - accuracy: 0.9632 - val_loss: 1.2842 - val_accuracy: 0.7430 - lr: 1.0000e-05 Epoch 71/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1089 - accuracy: 0.9602 - val_loss: 1.2643 - val_accuracy: 0.7450 - lr: 1.0000e-05 Epoch 72/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1109 - accuracy: 0.9591 - val_loss: 1.2686 - val_accuracy: 0.7517 - lr: 1.0000e-05 Epoch 73/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1053 - accuracy: 0.9628 - val_loss: 1.2550 - val_accuracy: 0.7428 - lr: 1.0000e-05 Epoch 74/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1052 - accuracy: 0.9600 - val_loss: 1.2497 - val_accuracy: 0.7456 - lr: 1.0000e-05 Epoch 75/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1007 - accuracy: 0.9621 - val_loss: 1.2674 - val_accuracy: 0.7392 - lr: 1.0000e-05
plot_model_performance_and_confusion_matrix(cnn_model_4, history, test_generator,
metrics_function=lambda true, pred: print(classification_report(true, pred)))
from IPython.display import HTML, display
val_loss, val_accuracy = cnn_model_4.evaluate(
test_generator,
steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
# Format the evaluation results as HTML
output_html = f"""
<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>
<p style='color: red;'><b>Test Loss:</b> CNN_MODEL_4 </p>
<p style='color: red;'><b>Test Loss:</b> {val_loss:.4f}</p>
<p style='color: blue;'><b>Test Accuracy:</b> {val_accuracy:.4%} </p>
</div>
"""
# Display the styled output
display(HTML(output_html))
4/4 [==============================] - 0s 17ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
precision recall f1-score support
0 0.86 0.78 0.82 32
1 0.65 0.75 0.70 32
2 0.70 0.66 0.68 32
3 0.84 0.84 0.84 32
accuracy 0.76 128
macro avg 0.76 0.76 0.76 128
weighted avg 0.76 0.76 0.76 128
4/4 [==============================] - 0s 20ms/step - loss: 0.6133 - accuracy: 0.7578 Test Loss: 0.6132991909980774 Test Accuracy: 0.7578125
Test Loss: CNN_MODEL_4
Test Loss: 0.6133
Test Accuracy: 75.7812%
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix
predicted_labels = np.argmax(cnn_model_4.predict(test_generator), axis=1)
# correcting mislabeling
true_labels = test_generator.classes
# Get the class names
class_names_list = test_generator.class_indices.keys()
print(class_names_list)
# Calculate and display the metrics
metrics_score(true_labels, predicted_labels)
# Plot the confusion matrix
cm = confusion_matrix(true_labels, predicted_labels)
4/4 [==============================] - 0s 17ms/step
dict_keys(['happy', 'neutral', 'sad', 'surprise'])
precision recall f1-score support
0 0.96 0.81 0.88 32
1 0.60 0.81 0.69 32
2 0.71 0.69 0.70 32
3 1.00 0.84 0.92 32
accuracy 0.79 128
macro avg 0.82 0.79 0.80 128
weighted avg 0.82 0.79 0.80 128
It is then used to perform feature extraction on the images in the pipeline
the Siamese network outputs the features from the last layer before the final similarity score calculation.
Then those output feature vectors are used as the inputs to the CNN_MODEL_5
Start with defining the input shape for our images, making sure it matches the resolution we're working with, like (48, 48, 3) for 48x48 RGB images.
Create the base CNN model that learns feature representations from the images. This model is crucial as it captures the essence of the images.
Define two input tensors for the Siamese network, representing the pair of images to compare.
Reuse the base CNN model for both inputs to ensure the feature extraction process is identical for each image in the pair.
Process both inputs through the base CNN independently, obtaining their feature representations.
Calculate the distance between the two sets of features to understand how similar or dissimilar the images are. The absolute difference works well for this.
Feed the distance through a dense layer with a sigmoid activation to get a similarity score, indicating the likelihood of the images being the same class. Compile the Siamese network, choosing an optimizer and a loss function tailored for comparison tasks, like contrastive loss.
Prepare the image pairs for training, ensuring we have both positive pairs (same class) and negative pairs (different classes).
Augment the dataset if needed, especially to address class imbalance or to increase the diversity of training examples.
Train the Siamese network on these pairs, focusing on learning the subtle differences and similarities between classes.
Evaluate the model on a separate validation set to ensure it generalizes well to new, unseen image pairs.
Siamese for Feature Vectors: I use the trained Siamese model, or specifically its base CNN part, to extract feature vectors from images. This involves modifying the Siamese network to output features from the layer just before the final similarity score calculation.
CNN for Classification: The extracted feature vectors serve as inputs to cnn_model_4 or a similar classification model. This requires me to adjust cnn_model_4 to accept the feature vector size as its input shape, possibly changing the first layer to a Dense layer from a Conv2D, or using the features directly in a custom training loop to influence the classification decision. Using Siamese Model for Data Augmentation:
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=False,
fill_mode='nearest'
)
# Define the image size and paths
image_size = 48
train_path = "Facial_emotion_images/train"
test_path = "Facial_emotion_images/test"
validation_path = "Facial_emotion_images/validation"
# Initialize the data generator with rescaling
datagen = ImageDataGenerator(rescale=1./255)
# Prepare the data generators
train_generator = datagen.flow_from_directory(
train_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical')
validation_generator = datagen.flow_from_directory(
validation_path,
target_size=(48, 48),
batch_size=32, # Keep the same batch size for simplicity
class_mode='categorical')
test_generator = datagen.flow_from_directory(
test_path,
target_size=(48, 48),
shuffle=False)
Found 15109 images belonging to 4 classes. Found 4977 images belonging to 4 classes. Found 128 images belonging to 4 classes.
cnn_model_4 = Sequential([
Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(48, 48, 3)),
BatchNormalization(),
Conv2D(64, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.3), # Slightly reduced dropout
# Second Convolutional Block
Conv2D(128, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(256, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.3), # Adjusted dropout
# Third Convolutional Block
Conv2D(256, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(512, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.4), # Adjusted dropout
# Fourth Convolutional Block
Conv2D(1024, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.2), # Keeping consistent dropout
# Convolutional Block 5
Conv2D(1024, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.2), # Keeping consistent dropout
# Global Average Pooling replaces Flatten to reduce parameters and potential overfitting
GlobalAveragePooling2D(),
Flatten(),
# Dense Layers
Dense(128, activation='relu'),
Dense(512, activation='relu'),
Dropout(0.2),
Dense(1024, activation='relu'),
Dense(2048, activation='relu'),
Dropout(0.4),
Dense(128, activation='relu'),
Dropout(0.2),
Dense(4, activation='softmax')
])
cnn_model_4.compile(optimizer=AdamW(learning_rate=0.00003),
loss='categorical_crossentropy',
metrics=['accuracy'])
early_stopping = EarlyStopping(monitor='val_loss', patience=10, verbose=1, restore_best_weights=True) # Increased patience
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, verbose=1, min_lr=0.00001) # Adjusted patience
cnn_model_4.summary()
Model: "sequential_43"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_210 (Conv2D) (None, 48, 48, 64) 1792
batch_normalization_148 (B (None, 48, 48, 64) 256
atchNormalization)
conv2d_211 (Conv2D) (None, 48, 48, 64) 36928
max_pooling2d_123 (MaxPool (None, 24, 24, 64) 0
ing2D)
dropout_178 (Dropout) (None, 24, 24, 64) 0
conv2d_212 (Conv2D) (None, 24, 24, 128) 73856
batch_normalization_149 (B (None, 24, 24, 128) 512
atchNormalization)
conv2d_213 (Conv2D) (None, 24, 24, 256) 295168
max_pooling2d_124 (MaxPool (None, 12, 12, 256) 0
ing2D)
dropout_179 (Dropout) (None, 12, 12, 256) 0
conv2d_214 (Conv2D) (None, 12, 12, 256) 590080
batch_normalization_150 (B (None, 12, 12, 256) 1024
atchNormalization)
conv2d_215 (Conv2D) (None, 12, 12, 512) 1180160
batch_normalization_151 (B (None, 12, 12, 512) 2048
atchNormalization)
max_pooling2d_125 (MaxPool (None, 6, 6, 512) 0
ing2D)
dropout_180 (Dropout) (None, 6, 6, 512) 0
conv2d_216 (Conv2D) (None, 6, 6, 1024) 4719616
batch_normalization_152 (B (None, 6, 6, 1024) 4096
atchNormalization)
max_pooling2d_126 (MaxPool (None, 3, 3, 1024) 0
ing2D)
dropout_181 (Dropout) (None, 3, 3, 1024) 0
conv2d_217 (Conv2D) (None, 3, 3, 1024) 9438208
batch_normalization_153 (B (None, 3, 3, 1024) 4096
atchNormalization)
max_pooling2d_127 (MaxPool (None, 1, 1, 1024) 0
ing2D)
dropout_182 (Dropout) (None, 1, 1, 1024) 0
global_average_pooling2d_3 (None, 1024) 0
0 (GlobalAveragePooling2D)
flatten_30 (Flatten) (None, 1024) 0
dense_146 (Dense) (None, 128) 131200
dense_147 (Dense) (None, 512) 66048
dropout_183 (Dropout) (None, 512) 0
dense_148 (Dense) (None, 1024) 525312
dense_149 (Dense) (None, 2048) 2099200
dropout_184 (Dropout) (None, 2048) 0
dense_150 (Dense) (None, 128) 262272
dropout_185 (Dropout) (None, 128) 0
dense_151 (Dense) (None, 4) 516
=================================================================
Total params: 19432388 (74.13 MB)
Trainable params: 19426372 (74.11 MB)
Non-trainable params: 6016 (23.50 KB)
_________________________________________________________________
input_shape = (48, 48, 3)
base_cnn = create_base_cnn(input_shape)
feature_model = Model(inputs=base_cnn.input, outputs=base_cnn.layers[-5].output)
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dropout, GlobalAveragePooling2D, Dense, BatchNormalization, Flatten
from tensorflow.keras.regularizers import l2
def create_base_cnn(input_shape):
model = Sequential([
Conv2D(64, (3, 3), activation='relu', input_shape=input_shape, padding='same'),
BatchNormalization(),
Conv2D(64, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.3),
Conv2D(128, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(256, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.3),
Conv2D(256, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(512, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.4),
GlobalAveragePooling2D(),
Dense(512, activation='relu'),
Dropout(0.4),
Dense(512, activation='relu', kernel_regularizer=l2(0.001)),
])
return model
from tensorflow.keras.layers import Input, Lambda
from tensorflow.keras.models import Model
import tensorflow.keras.backend as K
input_shape = (48, 48, 3)
def create_siamese_network(input_shape):
input_a = Input(shape=input_shape)
input_b = Input(shape=input_shape)
base_cnn = create_base_cnn(input_shape)
processed_a = base_cnn(input_a)
processed_b = base_cnn(input_b)
distance = Lambda(lambda tensors: K.abs(tensors[0] - tensors[1]))([processed_a, processed_b])
outputs = Dense(1, activation='sigmoid')(distance)
model = Model(inputs=[input_a, input_b], outputs=outputs)
return model
import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Conv2D, MaxPooling2D, Flatten, Dense, Lambda
from tensorflow.keras.optimizers import Adam
import tensorflow.keras.backend as K
from sklearn.model_selection import train_test_split
def create_base_cnn(input_shape):
input = Input(shape=input_shape)
x = Conv2D(64, (3, 3), activation='relu')(input)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Conv2D(128, (3, 3), activation='relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Flatten()(x)
x = Dense(128, activation='relu')(x)
return Model(inputs=input, outputs=x)
def create_siamese_network(input_shape):
base_network = create_base_cnn(input_shape)
input_a = Input(shape=input_shape)
input_b = Input(shape=input_shape)
processed_a = base_network(input_a)
processed_b = base_network(input_b)
distance = Lambda(lambda tensors: K.abs(tensors[0] - tensors[1]))([processed_a, processed_b])
outputs = Dense(1, activation='sigmoid')(distance)
model = Model(inputs=[input_a, input_b], outputs=outputs)
return model
input_shape = (48, 48, 3) # Adjust based on your dataset
siamese_network = create_siamese_network(input_shape)
siamese_network.compile(optimizer=Adam(0.0001), loss='binary_crossentropy', metrics=['accuracy'])
# Define callbacks
early_stopping = EarlyStopping(monitor='val_loss', patience=10, verbose=1, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, verbose=1, min_lr=0.00001)
# Fit the model
history = cnn_model_4.fit(
train_generator,
steps_per_epoch=train_generator.samples // train_generator.batch_size,
epochs=100,
validation_data=validation_generator,# Adjusted data loading
validation_steps=validation_generator.samples // validation_generator.batch_size,# adjusted data loading
callbacks=[early_stopping, reduce_lr],
verbose=1
)
Epoch 1/100 472/472 [==============================] - 25s 43ms/step - loss: 1.3865 - accuracy: 0.3460 - val_loss: 1.6837 - val_accuracy: 0.3663 - lr: 3.0000e-05 Epoch 2/100 472/472 [==============================] - 20s 43ms/step - loss: 1.2545 - accuracy: 0.4210 - val_loss: 1.5015 - val_accuracy: 0.3835 - lr: 3.0000e-05 Epoch 3/100 472/472 [==============================] - 20s 43ms/step - loss: 1.1560 - accuracy: 0.4819 - val_loss: 1.3127 - val_accuracy: 0.4194 - lr: 3.0000e-05 Epoch 4/100 472/472 [==============================] - 20s 43ms/step - loss: 1.0680 - accuracy: 0.5405 - val_loss: 1.1432 - val_accuracy: 0.5143 - lr: 3.0000e-05 Epoch 5/100 472/472 [==============================] - 20s 42ms/step - loss: 1.0107 - accuracy: 0.5706 - val_loss: 0.9674 - val_accuracy: 0.5817 - lr: 3.0000e-05 Epoch 6/100 472/472 [==============================] - 20s 42ms/step - loss: 0.9745 - accuracy: 0.5841 - val_loss: 1.1597 - val_accuracy: 0.5379 - lr: 3.0000e-05 Epoch 7/100 472/472 [==============================] - 20s 43ms/step - loss: 0.9306 - accuracy: 0.6095 - val_loss: 0.9213 - val_accuracy: 0.6222 - lr: 3.0000e-05 Epoch 8/100 472/472 [==============================] - 20s 43ms/step - loss: 0.8935 - accuracy: 0.6266 - val_loss: 0.8629 - val_accuracy: 0.6468 - lr: 3.0000e-05 Epoch 9/100 472/472 [==============================] - 20s 43ms/step - loss: 0.8591 - accuracy: 0.6397 - val_loss: 0.9322 - val_accuracy: 0.6288 - lr: 3.0000e-05 Epoch 10/100 472/472 [==============================] - 20s 42ms/step - loss: 0.8364 - accuracy: 0.6540 - val_loss: 0.8387 - val_accuracy: 0.6633 - lr: 3.0000e-05 Epoch 11/100 472/472 [==============================] - 20s 43ms/step - loss: 0.8130 - accuracy: 0.6615 - val_loss: 0.8049 - val_accuracy: 0.6722 - lr: 3.0000e-05 Epoch 12/100 472/472 [==============================] - 20s 42ms/step - loss: 0.7941 - accuracy: 0.6763 - val_loss: 0.7672 - val_accuracy: 0.7006 - lr: 3.0000e-05 Epoch 13/100 472/472 [==============================] - 20s 43ms/step - loss: 0.7713 - accuracy: 0.6821 - val_loss: 0.7706 - val_accuracy: 0.6976 - lr: 3.0000e-05 Epoch 14/100 472/472 [==============================] - 20s 43ms/step - loss: 0.7454 - accuracy: 0.6914 - val_loss: 0.7470 - val_accuracy: 0.7002 - lr: 3.0000e-05 Epoch 15/100 472/472 [==============================] - 20s 43ms/step - loss: 0.7268 - accuracy: 0.7003 - val_loss: 0.7572 - val_accuracy: 0.6978 - lr: 3.0000e-05 Epoch 16/100 472/472 [==============================] - 20s 43ms/step - loss: 0.7109 - accuracy: 0.7088 - val_loss: 0.7452 - val_accuracy: 0.7026 - lr: 3.0000e-05 Epoch 17/100 472/472 [==============================] - 20s 43ms/step - loss: 0.6928 - accuracy: 0.7184 - val_loss: 0.7401 - val_accuracy: 0.7079 - lr: 3.0000e-05 Epoch 18/100 472/472 [==============================] - 20s 43ms/step - loss: 0.6757 - accuracy: 0.7249 - val_loss: 0.7120 - val_accuracy: 0.7216 - lr: 3.0000e-05 Epoch 19/100 472/472 [==============================] - 21s 44ms/step - loss: 0.6559 - accuracy: 0.7340 - val_loss: 0.6963 - val_accuracy: 0.7278 - lr: 3.0000e-05 Epoch 20/100 472/472 [==============================] - 20s 43ms/step - loss: 0.6376 - accuracy: 0.7405 - val_loss: 0.6986 - val_accuracy: 0.7262 - lr: 3.0000e-05 Epoch 21/100 472/472 [==============================] - 20s 43ms/step - loss: 0.6212 - accuracy: 0.7464 - val_loss: 0.7125 - val_accuracy: 0.7254 - lr: 3.0000e-05 Epoch 22/100 472/472 [==============================] - 20s 43ms/step - loss: 0.6017 - accuracy: 0.7613 - val_loss: 0.6822 - val_accuracy: 0.7323 - lr: 3.0000e-05 Epoch 23/100 472/472 [==============================] - 20s 43ms/step - loss: 0.5885 - accuracy: 0.7675 - val_loss: 0.6721 - val_accuracy: 0.7397 - lr: 3.0000e-05 Epoch 24/100 472/472 [==============================] - 20s 43ms/step - loss: 0.5641 - accuracy: 0.7730 - val_loss: 0.6693 - val_accuracy: 0.7454 - lr: 3.0000e-05 Epoch 25/100 472/472 [==============================] - 20s 43ms/step - loss: 0.5524 - accuracy: 0.7795 - val_loss: 0.6849 - val_accuracy: 0.7361 - lr: 3.0000e-05 Epoch 26/100 472/472 [==============================] - 20s 43ms/step - loss: 0.5410 - accuracy: 0.7822 - val_loss: 0.6808 - val_accuracy: 0.7345 - lr: 3.0000e-05 Epoch 27/100 472/472 [==============================] - 20s 43ms/step - loss: 0.5218 - accuracy: 0.7951 - val_loss: 0.6800 - val_accuracy: 0.7399 - lr: 3.0000e-05 Epoch 28/100 472/472 [==============================] - 20s 43ms/step - loss: 0.5034 - accuracy: 0.8005 - val_loss: 0.6799 - val_accuracy: 0.7407 - lr: 3.0000e-05 Epoch 29/100 472/472 [==============================] - 20s 43ms/step - loss: 0.4944 - accuracy: 0.8042 - val_loss: 0.6570 - val_accuracy: 0.7532 - lr: 3.0000e-05 Epoch 30/100 472/472 [==============================] - 20s 43ms/step - loss: 0.4700 - accuracy: 0.8151 - val_loss: 0.6590 - val_accuracy: 0.7516 - lr: 3.0000e-05 Epoch 31/100 472/472 [==============================] - 20s 43ms/step - loss: 0.4588 - accuracy: 0.8213 - val_loss: 0.6479 - val_accuracy: 0.7506 - lr: 3.0000e-05 Epoch 32/100 472/472 [==============================] - 20s 43ms/step - loss: 0.4400 - accuracy: 0.8301 - val_loss: 0.6796 - val_accuracy: 0.7468 - lr: 3.0000e-05 Epoch 33/100 472/472 [==============================] - 20s 43ms/step - loss: 0.4303 - accuracy: 0.8328 - val_loss: 0.6759 - val_accuracy: 0.7433 - lr: 3.0000e-05 Epoch 34/100 472/472 [==============================] - 20s 43ms/step - loss: 0.4136 - accuracy: 0.8395 - val_loss: 0.6724 - val_accuracy: 0.7496 - lr: 3.0000e-05 Epoch 35/100 472/472 [==============================] - 20s 43ms/step - loss: 0.4050 - accuracy: 0.8461 - val_loss: 0.7359 - val_accuracy: 0.7252 - lr: 3.0000e-05 Epoch 36/100 471/472 [============================>.] - ETA: 0s - loss: 0.3864 - accuracy: 0.8512 Epoch 36: ReduceLROnPlateau reducing learning rate to 1e-05. 472/472 [==============================] - 20s 43ms/step - loss: 0.3865 - accuracy: 0.8512 - val_loss: 0.6757 - val_accuracy: 0.7506 - lr: 3.0000e-05 Epoch 37/100 472/472 [==============================] - 20s 43ms/step - loss: 0.3486 - accuracy: 0.8706 - val_loss: 0.6466 - val_accuracy: 0.7641 - lr: 1.0000e-05 Epoch 38/100 472/472 [==============================] - 20s 43ms/step - loss: 0.3408 - accuracy: 0.8740 - val_loss: 0.6653 - val_accuracy: 0.7571 - lr: 1.0000e-05 Epoch 39/100 472/472 [==============================] - 20s 43ms/step - loss: 0.3252 - accuracy: 0.8841 - val_loss: 0.6793 - val_accuracy: 0.7526 - lr: 1.0000e-05 Epoch 40/100 472/472 [==============================] - 20s 43ms/step - loss: 0.3270 - accuracy: 0.8804 - val_loss: 0.6592 - val_accuracy: 0.7647 - lr: 1.0000e-05 Epoch 41/100 472/472 [==============================] - 20s 43ms/step - loss: 0.3145 - accuracy: 0.8842 - val_loss: 0.6528 - val_accuracy: 0.7663 - lr: 1.0000e-05 Epoch 42/100 472/472 [==============================] - 20s 43ms/step - loss: 0.3107 - accuracy: 0.8898 - val_loss: 0.6633 - val_accuracy: 0.7649 - lr: 1.0000e-05 Epoch 43/100 472/472 [==============================] - 20s 43ms/step - loss: 0.3040 - accuracy: 0.8900 - val_loss: 0.6761 - val_accuracy: 0.7603 - lr: 1.0000e-05 Epoch 44/100 472/472 [==============================] - 20s 43ms/step - loss: 0.3054 - accuracy: 0.8869 - val_loss: 0.6615 - val_accuracy: 0.7637 - lr: 1.0000e-05 Epoch 45/100 472/472 [==============================] - 20s 43ms/step - loss: 0.2913 - accuracy: 0.8928 - val_loss: 0.6561 - val_accuracy: 0.7681 - lr: 1.0000e-05 Epoch 46/100 472/472 [==============================] - 20s 43ms/step - loss: 0.2858 - accuracy: 0.8963 - val_loss: 0.6878 - val_accuracy: 0.7579 - lr: 1.0000e-05 Epoch 47/100 471/472 [============================>.] - ETA: 0s - loss: 0.2852 - accuracy: 0.8956Restoring model weights from the end of the best epoch: 37. 472/472 [==============================] - 20s 43ms/step - loss: 0.2857 - accuracy: 0.8954 - val_loss: 0.6985 - val_accuracy: 0.7550 - lr: 1.0000e-05 Epoch 47: early stopping
# Evaluate the model on the test set
val_loss, val_accuracy = cnn_model_4.evaluate(
test_generator,
steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
4/4 [==============================] - 0s 16ms/step - loss: 0.5081 - accuracy: 0.7969 Test Loss: 0.5081095695495605 Test Accuracy: 0.796875
from sklearn.metrics import confusion_matrix, classification_report
# Plot the training and validation accuracy values
plt.figure(figsize=(35, 6))
plt.subplot(1, 4, 1)
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('cnn_model_4 Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Plot the training and validation loss values
plt.subplot(1, 4, 2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('cnn_model_4 Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Plot the training and validation loss values
plt.subplot(1, 4, 3)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", cbar=False)
plt.xlabel("Predicted Label")
plt.ylabel("True Label")
plt.title("Confusion Matrix for cnn_model_4")
predicted_labels = np.argmax(cnn_model_4.predict(test_generator), axis=1)
# Get the true labels for the test set
true_labels = test_generator.classes
# Get the class names
class_names_list = test_generator.class_indices.keys()
# Calculate and display the metrics
metrics_score(true_labels, predicted_labels)
plt.show()
4/4 [==============================] - 0s 16ms/step
precision recall f1-score support
0 0.83 0.91 0.87 32
1 0.69 0.75 0.72 32
2 0.72 0.66 0.69 32
3 0.97 0.88 0.92 32
accuracy 0.80 128
macro avg 0.80 0.80 0.80 128
weighted avg 0.80 0.80 0.80 128
!tensorflowjs_converter --input_format=keras /content/cnn_model_4.h5 /content/siamese_model_1
2024-02-22 05:12:35.043733: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-02-22 05:12:35.043782: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-02-22 05:12:35.044911: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-02-22 05:12:36.086643: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
However, the methods used show that substantial gains can be made:
Data augmentation for the training data helps to make the smaller dataset more robust. Weighting the classes helped with making the class distribution skew less detrimental to the overall performance. Transfer learning on the dataset proved to be difficult; however, after a good amount of hyperparameter adjusting, two of the three transfer learning models started to show promising results. The second convolutional neural network constructed is the overall best performer among the models built here.
The final model tries to extract useful feature vector data and then fuse its learned properties with the training process of a standard CNN. This is an attempt to take a more specialized method to really learn and embed the deep underlying properties that differentiate the classes of neutral and sad. Those two classes gave all the models the most difficulty during testing, as seen in the included confusion matrices for each model's testing results.
This task is important. Being able to detect basic emotions rests upon the base technologies that will enable more impactful use cases, such as driver monitoring in automobiles, to check if the driver is paying attention. This is especially important with level 2 and 3 semi-autonomous vehicles where there is a notable vigilance decrement by drivers. Drivers can get the sense that the car is fully in control and will start not only to not pay attention but, in some extreme cases, even fall asleep. The methods to be able to detect a human face and determine the emotion of an individual can easily be extended and used to track the driver's attention span and ensure that they are ready to take back control of the vehicle.
In this project, we developed a facial recognition and emotion classification model using a convolutional neural network (CNN). The model, a 5-layer CNN with deep dense layers and optimized convolution setup, achieved 82% accuracy.
The main challenge was accurately classifying facial emotions due to the subtle differences between expressions. This is critical for sectors like security and customer service, where understanding human emotions is key.
The chosen CNN architecture relys on deep learning's pattern recognition capabilities, especially for image data, enhancing the model's ability to detect subtle emotional features. Implementing this model could significantly improve human-computer interactions, making them more responsive and empathetic. For customer service, it could lead to tailored communications and solutions, boosting satisfaction and loyalty. In security, it could help identify distress or suspicious behaviors more effectively. Recommendations for Implementation Integration with Interactive Systems: Deploy the model in platforms for enhanced user interaction, such as customer service kiosks or online help desks. Enhancement of Security Systems: Use the model for improved surveillance and monitoring, flagging unusual or suspicious behavior. Actionables for Stakeholders:
Deployment and Training: Implement the model on targeted platforms and train staff to use it effectively. Continuous Model Evaluation and Improvement: Regularly review and update the model with new data to ensure its accuracy and relevance. Privacy and Ethical Considerations: Set guidelines for ethical use, respecting privacy and data protection laws.
Costs: The training of these models required backend gpu services provided through google. For a larger scale version the costs would go up accordingly. This project alone used 200 compute units over the period of a week training and re-training models. For a more complex model that would be required to scale up the services explored here cost analysis should be taken into consideration depending on the rate of scaling.
Available: 16.86 compute units Usage rate: approximately 2.05 per hour
To conclude, Computer Vision, is where all the code and software actually make contact with our world. Unlike robotics, for computer vision all that is required is a simple camera, giving the technology the ability to reach and touch most of the population on the planet depening on the use case.
Thank you -- Robert A, Sloan
plot_model_performance_and_confusion_matrix(cnn_model_1, history, test_generator,
metrics_function=lambda true, pred: print(classification_report(true, pred)))
4/4 [==============================] - 0s 15ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
precision recall f1-score support
0 0.87 0.81 0.84 32
1 0.65 0.88 0.75 32
2 0.81 0.69 0.75 32
3 0.96 0.84 0.90 32
accuracy 0.80 128
macro avg 0.82 0.80 0.81 128
weighted avg 0.82 0.80 0.81 128
4/4 [==============================] - 0s 10ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
precision recall f1-score support
0 0.82 0.84 0.83 32
1 0.72 0.81 0.76 32
2 0.79 0.69 0.73 32
3 0.97 0.94 0.95 32
accuracy 0.82 128
macro avg 0.82 0.82 0.82 128
weighted avg 0.82 0.82 0.82 128
4/4 [==============================] - 0s 11ms/step - loss: 0.5187 - accuracy: 0.8203 Test Loss: 0.5187498927116394 Test Accuracy: 0.8203125
Test Loss: CNN_MODEL_updated
Test Loss: 0.5187
Test Accuracy: 82.0312%
4/4 [==============================] - 0s 11ms/step Confusion Matrix: [[25 6 0 1] [ 0 28 3 1] [ 3 8 18 3] [ 0 2 2 28]]
4/4 [==============================] - 0s 13ms/step
dict_keys(['happy', 'neutral', 'sad', 'surprise'])
precision recall f1-score support
0 0.89 0.78 0.83 32
1 0.64 0.88 0.74 32
2 0.78 0.56 0.65 32
3 0.85 0.88 0.86 32
accuracy 0.77 128
macro avg 0.79 0.77 0.77 128
weighted avg 0.79 0.77 0.77 128
4/4 [==============================] - 1s 20ms/step Confusion Matrix: [[26 2 2 2] [ 2 25 5 0] [ 1 12 17 2] [ 0 3 1 28]]
4/4 [==============================] - 0s 22ms/step
dict_keys(['happy', 'neutral', 'sad', 'surprise'])
precision recall f1-score support
0 0.90 0.81 0.85 32
1 0.60 0.78 0.68 32
2 0.68 0.53 0.60 32
3 0.88 0.88 0.88 32
accuracy 0.75 128
macro avg 0.76 0.75 0.75 128
weighted avg 0.76 0.75 0.75 128
4/4 [==============================] - 0s 12ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
precision recall f1-score support
0 0.67 0.81 0.73 32
1 0.57 0.72 0.64 32
2 0.54 0.44 0.48 32
3 0.87 0.62 0.73 32
accuracy 0.65 128
macro avg 0.66 0.65 0.65 128
weighted avg 0.66 0.65 0.65 128
4/4 [==============================] - 0s 18ms/step - loss: 0.8464 - accuracy: 0.6484 Test Loss: 0.84635990858078 Test Accuracy: 0.6484375
Test Loss: efficient_model2
Test Loss: 0.85
Test Accuracy: 64.84%
4/4 [==============================] - 0s 17ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
precision recall f1-score support
0 0.86 0.78 0.82 32
1 0.65 0.75 0.70 32
2 0.70 0.66 0.68 32
3 0.84 0.84 0.84 32
accuracy 0.76 128
macro avg 0.76 0.76 0.76 128
weighted avg 0.76 0.76 0.76 128
4/4 [==============================] - 0s 20ms/step - loss: 0.6133 - accuracy: 0.7578 Test Loss: 0.6132991909980774 Test Accuracy: 0.7578125
Test Loss: CNN_MODEL_4
Test Loss: 0.6133
Test Accuracy: 75.7812%
4/4 [==============================] - 0s 16ms/step
precision recall f1-score support
0 0.83 0.91 0.87 32
1 0.69 0.75 0.72 32
2 0.72 0.66 0.69 32
3 0.97 0.88 0.92 32
accuracy 0.80 128
macro avg 0.80 0.80 0.80 128
weighted avg 0.80 0.80 0.80 128
!tensorflowjs_converter --input_format=keras /content/cnn_model_4.h5 /content/siamese_model_1
2024-02-22 05:12:35.043733: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-02-22 05:12:35.043782: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-02-22 05:12:35.044911: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-02-22 05:12:36.086643: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
# prompt: using the nbconvert tool convert the notebook to html
!jupyter nbconvert --to html "/content/cnn_model_4.ipynb"
[NbConvertApp] WARNING | pattern '/content/notebook.ipynb' matched no files
This application is used to convert notebook files (*.ipynb)
to various other formats.
WARNING: THE COMMANDLINE INTERFACE MAY CHANGE IN FUTURE RELEASES.
Options
=======
The options below are convenience aliases to configurable class-options,
as listed in the "Equivalent to" description-line of the aliases.
To see all configurable class-options for some <cmd>, use:
<cmd> --help-all
--debug
set log level to logging.DEBUG (maximize logging output)
Equivalent to: [--Application.log_level=10]
--show-config
Show the application's configuration (human-readable format)
Equivalent to: [--Application.show_config=True]
--show-config-json
Show the application's configuration (json format)
Equivalent to: [--Application.show_config_json=True]
--generate-config
generate default config file
Equivalent to: [--JupyterApp.generate_config=True]
-y
Answer yes to any questions instead of prompting.
Equivalent to: [--JupyterApp.answer_yes=True]
--execute
Execute the notebook prior to export.
Equivalent to: [--ExecutePreprocessor.enabled=True]
--allow-errors
Continue notebook execution even if one of the cells throws an error and include the error message in the cell output (the default behaviour is to abort conversion). This flag is only relevant if '--execute' was specified, too.
Equivalent to: [--ExecutePreprocessor.allow_errors=True]
--stdin
read a single notebook file from stdin. Write the resulting notebook with default basename 'notebook.*'
Equivalent to: [--NbConvertApp.from_stdin=True]
--stdout
Write notebook output to stdout instead of files.
Equivalent to: [--NbConvertApp.writer_class=StdoutWriter]
--inplace
Run nbconvert in place, overwriting the existing notebook (only
relevant when converting to notebook format)
Equivalent to: [--NbConvertApp.use_output_suffix=False --NbConvertApp.export_format=notebook --FilesWriter.build_directory=]
--clear-output
Clear output of current file and save in place,
overwriting the existing notebook.
Equivalent to: [--NbConvertApp.use_output_suffix=False --NbConvertApp.export_format=notebook --FilesWriter.build_directory= --ClearOutputPreprocessor.enabled=True]
--no-prompt
Exclude input and output prompts from converted document.
Equivalent to: [--TemplateExporter.exclude_input_prompt=True --TemplateExporter.exclude_output_prompt=True]
--no-input
Exclude input cells and output prompts from converted document.
This mode is ideal for generating code-free reports.
Equivalent to: [--TemplateExporter.exclude_output_prompt=True --TemplateExporter.exclude_input=True --TemplateExporter.exclude_input_prompt=True]
--allow-chromium-download
Whether to allow downloading chromium if no suitable version is found on the system.
Equivalent to: [--WebPDFExporter.allow_chromium_download=True]
--disable-chromium-sandbox
Disable chromium security sandbox when converting to PDF..
Equivalent to: [--WebPDFExporter.disable_sandbox=True]
--show-input
Shows code input. This flag is only useful for dejavu users.
Equivalent to: [--TemplateExporter.exclude_input=False]
--embed-images
Embed the images as base64 dataurls in the output. This flag is only useful for the HTML/WebPDF/Slides exports.
Equivalent to: [--HTMLExporter.embed_images=True]
--sanitize-html
Whether the HTML in Markdown cells and cell outputs should be sanitized..
Equivalent to: [--HTMLExporter.sanitize_html=True]
--log-level=<Enum>
Set the log level by value or name.
Choices: any of [0, 10, 20, 30, 40, 50, 'DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']
Default: 30
Equivalent to: [--Application.log_level]
--config=<Unicode>
Full path of a config file.
Default: ''
Equivalent to: [--JupyterApp.config_file]
--to=<Unicode>
The export format to be used, either one of the built-in formats
['asciidoc', 'custom', 'html', 'latex', 'markdown', 'notebook', 'pdf', 'python', 'rst', 'script', 'slides', 'webpdf']
or a dotted object name that represents the import path for an
``Exporter`` class
Default: ''
Equivalent to: [--NbConvertApp.export_format]
--template=<Unicode>
Name of the template to use
Default: ''
Equivalent to: [--TemplateExporter.template_name]
--template-file=<Unicode>
Name of the template file to use
Default: None
Equivalent to: [--TemplateExporter.template_file]
--theme=<Unicode>
Template specific theme(e.g. the name of a JupyterLab CSS theme distributed
as prebuilt extension for the lab template)
Default: 'light'
Equivalent to: [--HTMLExporter.theme]
--sanitize_html=<Bool>
Whether the HTML in Markdown cells and cell outputs should be sanitized.This
should be set to True by nbviewer or similar tools.
Default: False
Equivalent to: [--HTMLExporter.sanitize_html]
--writer=<DottedObjectName>
Writer class used to write the
results of the conversion
Default: 'FilesWriter'
Equivalent to: [--NbConvertApp.writer_class]
--post=<DottedOrNone>
PostProcessor class used to write the
results of the conversion
Default: ''
Equivalent to: [--NbConvertApp.postprocessor_class]
--output=<Unicode>
overwrite base name use for output files.
can only be used when converting one notebook at a time.
Default: ''
Equivalent to: [--NbConvertApp.output_base]
--output-dir=<Unicode>
Directory to write output(s) to. Defaults
to output to the directory of each notebook. To recover
previous default behaviour (outputting to the current
working directory) use . as the flag value.
Default: ''
Equivalent to: [--FilesWriter.build_directory]
--reveal-prefix=<Unicode>
The URL prefix for reveal.js (version 3.x).
This defaults to the reveal CDN, but can be any url pointing to a copy
of reveal.js.
For speaker notes to work, this must be a relative path to a local
copy of reveal.js: e.g., "reveal.js".
If a relative path is given, it must be a subdirectory of the
current directory (from which the server is run).
See the usage documentation
(https://nbconvert.readthedocs.io/en/latest/usage.html#reveal-js-html-slideshow)
for more details.
Default: ''
Equivalent to: [--SlidesExporter.reveal_url_prefix]
--nbformat=<Enum>
The nbformat version to write.
Use this to downgrade notebooks.
Choices: any of [1, 2, 3, 4]
Default: 4
Equivalent to: [--NotebookExporter.nbformat_version]
Examples
--------
The simplest way to use nbconvert is
> jupyter nbconvert mynotebook.ipynb --to html
Options include ['asciidoc', 'custom', 'html', 'latex', 'markdown', 'notebook', 'pdf', 'python', 'rst', 'script', 'slides', 'webpdf'].
> jupyter nbconvert --to latex mynotebook.ipynb
Both HTML and LaTeX support multiple output templates. LaTeX includes
'base', 'article' and 'report'. HTML includes 'basic', 'lab' and
'classic'. You can specify the flavor of the format used.
> jupyter nbconvert --to html --template lab mynotebook.ipynb
You can also pipe the output to stdout, rather than a file
> jupyter nbconvert mynotebook.ipynb --stdout
PDF is generated via latex
> jupyter nbconvert mynotebook.ipynb --to pdf
You can get (and serve) a Reveal.js-powered slideshow
> jupyter nbconvert myslides.ipynb --to slides --post serve
Multiple notebooks can be given at the command line in a couple of
different ways:
> jupyter nbconvert notebook*.ipynb
> jupyter nbconvert notebook1.ipynb notebook2.ipynb
or you can specify the notebooks list in a config file, containing::
c.NbConvertApp.notebooks = ["my_notebook.ipynb"]
> jupyter nbconvert --config mycfg.py
To see all available configurables, use `--help-all`.